Improve Your E-Mail Inbox Deliverability Rate

[Varius]

In this installment of the "GFY Educational Series", I'd like to touch on a topic that is critical to many businesses, but often misunderstood. The information is out there, but it takes quite a bit of digging to put the pieces together. It's my hope that I can save some of you that work and improve your bottom line with a few simple changes in how you manage your mailing campaigns.

This article is meant for legit, compliant mailing and not those looking to simply engage in mass spamming. This article will touch on mostly technical aspects of mailing and not approach it from a marketing sense, as we have already had an excellent Series article regarding that aspect.

1) IP Addresses

The first thing you need to do, above and beyond all else, is understand your current IP address and its history. You may have ordered a new server at your current host, moved to a new host or been guaranteed to have a "fresh" IP. You should not just take anyone's word for this; verify those claims yourself. There are numerous free tools to investigate IP addresses, so I'll let you find your favorite, but here is a checklist of what you are looking for:

- IP found on Blacklist(s). If your IP is found on a Blacklist and was a new IP given to you by your host, demand they give you a different one. Most Blacklists don't answer remove requests promptly (or at all). If this is an IP address you have been using for awhile though, as you'll see later in this article, it may have benefits to trying to "clean" it as opposed to starting from scratch with a new IP.

- IP in "Neighbourhood" found on Blacklist(s). If there is an IP on your C Class that is allocated to another client of your host who has a bad reputation for their mailing practices, this may affect your delivery. As above, get yourself a new IP on a different Class or possibly even change hosts (as you don't want to be associated with a host who is lenient towards spammers).

Except in rare cases, you should also not rotate IPs when you send mail out. This may work short-term, but ultimately causes you more hassle than it's worth long-term as well as preventing you from being granted whitelist status at many ISPs. If your volume of mail requires multiple dedicated mailservers, so be it, but try and limit this as much as possible. If you have several load-balanced webservers, simply have them relay their mail through a single dedicated mailserver (or cluster if absolutely needed). For redundancy, you should always have a backup mailserver ready to take over the original's IP and MAC address to takeover operations.

I also recommend that for separate websites, you use separate IP addresses when possible. This is akin to "not putting all your eggs in one basket". If one IP or site gets blocked, your other sites are not immediately "guilty by association".

Once you have verified your mailserver's IP address is clean, you are not in a bad neighbourhood and you are not rotating dozens of IP addresses, you have completed Step1

I recommend you setup regular monitoring of your IP address' reputation (either manual or automated), so you can be alerted if your situation ever changes and not let days go by wondering why your conversions/sales have dropped.


2) Reverse DNS

While this is an extremely simple point, many large sites seem to neglect it. Put simply, you want the IP address of the server your mail is sent from, to resolve to the mailserver's hostname. For example, if my mailserver's IP is 123.456.123.211 and it's seen by the outside world as "mail1.mydomain.com", then you want that anybody looking up that IP address sees it resolve to "mail1.domain.com".

Ususally, you don't own your own IPs, so you can simply ask your host to setup any reverse records you require. If you do own your IPs, you must set these up at your nameservers.


3) Bounce Management

While most sites employ valid unsubscribe features, very few utilize bounce management. There are many reasons an email will bounce and many of them are not your fault. However, ISPs pay heavy attention to who manages their bounces and who doesn't. In fact, a company like Yahoo! will not grant your whitelist request unless you can demonstrate proper bounce management.

There are multiple reasons for bounces, but they generally fall into 2 categories: soft bounces and hard bounces. A soft bounce is usually a temporary problem, such as a user is over their allowed quota (their mailbox is full). Hard bounces are permanent failures, such as emailing a domain which doesn't exist.

Whether you build or buy an application to handle bounces automatically, or have someone manually handling them, this is not something that should go on ignored.

If you receive a hard bounce, you should remove/block the email address immediately. Do not send to it again. If you receive a soft bounce, how you handle it is up to you, but generally I would block an email after receiving 3+ soft bounces over a 48+ hour period.

CONTINUED IN NEXT POST...

[Varius]

4) Mail Volume and Sending Habits

If you employ proper practices, it doesn't matter if you are sending out 1,000 or 1,000,000 emails a day. What matters is that you are CONSISTENT. If you send out a daily mailer, try to make sure it is approximately the same size and sent at approximately the same time every day. Obviously, as your site grows your volume should increase day to day; this is not a problem. A problem would be, on Monday you send out 500 emails, on Tuesday you send out 450,000 emails, on Wednesday you send another 250,000 and on Thursday you send 0. If you start mailing daily, be prepared to continue; stopping for multiple days in a row is a major red flag to ISPs. If you send out weekly or monthly newsletters, that is fine, it will just take said ISPs longer to recognize your sending patterns.

Aside from building up a reputation as a consistent sender, different ISPs have different limitations they don't always tell you about. Say your mail system is so strong, you could pound out 10 million emails an hour. If you check your mail log, you may notice a lot of "temporarily deferred" status messages. Yahoo and Hotmail in particular are two of the more stingy when it comes to rate-limiting. You will need to experiment for just what is the right frequency for each individual ISP, but you definitely should not be sending mail out as quick as you can. For example, Yahoo may like receiving only 15 emails from your server per second, while AOL is cool with you sending 200.

Once you have your habits in check, you can move on to the next step.


5) SPF Records / Sender-ID

Not too get too technical, but SPF records are a DNS record that is used by ISPs to determine if the sender is authorized to send mail out for this domain. This prevents source "spoofing" attempts (unless they can hack and modify your DNS record). This should be considered as an extra layer of security and not optional. Though not all ISPs use it, most of the majors do; so that makes it worth your time to implement. Microsoft has a specific version of the record they call "Sender-ID", but their wizard is often buggy so I would just generate it manually. SPF version 2 is also not yet complete, so in generating your record, you may wish to stick with "v=spf1" or specify both a version 1 and version 2 record.

If you need help generating your records, feel free to aks in the thread and I'll point you in the right direction.


6) DomainKeys / DKIM

Unlike Sender-ID, DomainKeys/DKIM are signing methods that you implement at a software level. Whether you use Qmail, Postfix or other, most of the major mailing softwares currently support it. DomainKeys has begun to fade away, to be replaced by DKIM, but there is no harm in signing your mail by both methods. Yahoo and Gmail are two solid examples of domainkeys; if you ever noticed a line in a message you received's header stating "this mail is signed bydomain.com", you have seen DomainKeys/DKIM at work.

As installation and configuration is different for every type of mailserver software, I won't get into the specifics of configuring it, however you will also need to add matching DNS records of type TXT in order for ISPs to verify your certificate against the authorized certificate on the domain. If this sounds a little too technical for you, it may be best to ask your host or hire a Server Admin to set this up for you.


7) Complaint Level

If you are collecting emails in a legitimate way, obeying unsubscribes and sending your members content that actually interests them, you should never have a complaint issue problem. Companies exist who offer monitoring tools, for a cost, if you are interested. Keep things above board and you will never have to worry about this factor.


8) Feedback Loops

Before many ISPs will even consider white-listing you, they will let you on their "Feedback Loop". This will forward you some complaints, from users who mark your mail as junk or spam as well as other remove requests/complaints. Pay attention to every mail they send you, take action when necessary and do it promptly. If you do that, ISPs will be willing to work with you to resolve problems.


9) Whitelist Status

If you have done all of the above and have been sending mail consistently for at least two or three months, you may apply for whitelist status at most of the major ISPs. The application is usually fairly detailed and may require some technical knowledge; be prepared for this. Once you have submitted an application, be patient and do not submit again. Within a few weeks, if you haven't heard anything back, you may try to submit again or contact their support. Once you are granted whitelist status, this does not guarantee that suddenly you will have a 100% Inbox rate; it will greatly, greatly improve your chances though. You may lose your status if you stop following the best practices outlined above.

CONTINUED IN NEXT POST...

[Varius]

10) SenderScore Certification

Microsoft's answer to a whitelist, getting SenderScore Certified is an often lengthy process and costs a decent amount of coin (depending on your volume). It was formerly known as Bonded Sending, where you have a pool of money that is substracted from with every complaint. If you follow the above guidelines, getting Certified should be fairly simple (but may take up to 90 days). This especially helps with Hotmail and other Microsoft products.


11) Mail Software Footprints

When using third-party software for managing your mail campaigns, always run tests and see what, if anything, they add to the header of a message. For example, many softwares like to place their "signature" into the header, typically as an X-Header: field. Some of these softwares, having been popular with spammers, draw red flags due to these signatures so turn them off when possible or research your software carefully before investing in it.


12) Open Relay

While most people are aware of this these days, by default, most mail agents allow open relaying and this is something you NEED to restrict. An open relay is a system that allows anyone to use it as an SMTP server for sending mail out; not just authorized users and addresses. Restricting it varies per mail software, but is generally quick and painless. Aside from the obvious problem of spammers finding and using your server, ruining your IP's reputation you worked so hard on keeping clean, ISPs can run a quick test to tell if you are "open" or not and if you are, they won't accept your mail in most cases.


13) Content / Filtering

Even if you pass all the above tests with flying colors, you still face a powerful foe; content filtering. This is more of a case-by-case issue, but I will attempt to give you a few tips.

First, know the major filtering softwares. For example, SpamAssassin is still a popular standard. Install them on your own if you have to and perform some trial and error, to determine the what words and email characteristics are heavy triggers and which don't carry much weight. Filter systems can check everything from "Do the domains in links in the email match the From: address?" to "what is the HTML/Text ratio versus images" along with weighing specific words, for example "viagra" is a commonly-blocked one.

Try and test your campaigns before sending them out for real, to get as low of a score as you can with different filtering systems. Again, this is a case-by-case basis factor and no one answer will magically get you through all filters.

One additional note: try and keep your HTML emails simple and clean (valid XHTML/CSS). Some ISPs are taking this into consideration.


14) Conclusion

There is still plenty more information on this topic, but the above points will hopefully have you understanding how to send emails out and how to reach your customers' inbox better than you currently do. Some industries, such as dating, rely on mail so much that a single day of being blocked can result in a conversion drop of up to 50%. The truth is, if your business relies heavily on email marketing, you should have someone dedicated to monitoring and fixing issues with your mail systems at least 2-3 hours per day.

Also, always remember that there are no guarantees in mailing. Unfortunately, due to abuse, some ISPs have become so tight that clients have a hard time receiving mail they want to receive! You may find yourself blocked for no apparent reason for a day or two, only to return to the inbox for another few months. Just do your best and you will see your deliverability increase.

If you have any specific questions, feel free to ask them in the thread below and I will answer them as best I can. I did not want to show any bias by recommending any particular monitoring or consulting services (as well as, I don't recommend anything I haven't used myself), but there are many out there who do genuinely know what they are doing. There are, of course, others who simply want your money and have no idea what they are doing.

I'll leave you with a handy tool, a method to easily test your SPF, Sender-ID, DomainKeys, DKIM and SpamAssassin score:

Send a test email out (ideally with the content you actually intend to send and from the real mailserver) to [email protected] and within minutes, they will reply to the From: address in your email with a report. Enjoy!


ABOUT THE AUTHOR

Varius (Keith) is best known around here for his many years in the adult dating space, being a part-owner of IwantU. He has regularly sent out millions of emails a day and spent a large amount of time understanding how to reach customers' inboxes. He currently is involved in multiple partnerships involving Adult, Gaming and Mainstream projects.

[LoveSandra]

nice reading..thnaks

[myjah]

Fantastic article - you've knocked it out of the park and I appreciate you stepping in and contributing on this.

[seeandsee]

just what i need, thanks a lot for this article!

[Mutt]

very informative on a subject i know zero about

[BestXXXPorn]

This is a fantastic intro to the variety of different key points you need to be taking into consideration when sending email. Great post Varius and, I'm sure, a welcome addition to the GFY Education series.

The only thing I would add is a bit of verbiage on content filtering and email clients... The big ones (Yahoo, Gmail, Live Mail, etc...) will weight your email based on content that is seemingly ok... e.g. Special Offer, Limited Time, Act Now, etc...

Great writeup, very solid

[RyuLion]

Wow dude! Epic thread!

[Varius]

Originally Posted by BestXXXPorn

The only thing I would add is a bit of verbiage on content filtering and email clients... The big ones (Yahoo, Gmail, Live Mail, etc...) will weight your email based on content that is seemingly ok... e.g. Special Offer, Limited Time, Act Now, etc...

Correct. A good general rule of thumb is try to stay away from phrases you find in spam you receive

Of course, if you have everything else done correctly, light-to-moderate usage of the above terms shouldn't result in any problems really.

[Lucky Bastard]

Fucking awesome article. Email (non)delivery is a nightmare of an issue.

[fuzebox]

Best article I've read in the series so far

[Varius]

Well, this fell off the first page fairly quick heh...I'll give it a bump for those who may have missed it

[munki]

Great write up... thanks!

[LeRoy]

Thats a lot of great info!

Thanks Varius

[cwd]

Thank you. Really good read.

[fatfoo]

Great article, Varius. I agree that spamming is not good, but legal mailing is fine.

[marketsmart]

Originally Posted by Varius

Well, this fell off the first page fairly quick heh...I'll give it a bump for those who may have missed it

i prefer to do it like the olden days....









.

[Thomas007]

Bump for good info.
I recently configured our new mail server, to have more control of what goes on in our end.

Some sources said that SPF is being obsolete if you use DKIM. So I chose not to setup SPF.

[Varius]

Originally Posted by Thomas007

Bump for good info.
I recently configured our new mail server, to have more control of what goes on in our end.

Some sources said that SPF is being obsolete if you use DKIM. So I chose not to setup SPF.

Not many ISPs use DKIM yet; you still need proper SPF records.

[Wizzo]

Good stuff Keith!

[CyberHustler]

Great thread... seriously.

[Jennyfer]

Great info!

[Varius]

Thank ye, thank ye - I'll be here all week. Or not, depending how fast this vanishes off the front page again

[craftyc]

Interesting article - why not go put it in articlebase and get yourself some exposure ;)

[d-null]

this is a great article on an often misunderstood subject, thanks for posting it

[Varius]

Originally Posted by craftyc

Interesting article - why not go put it in articlebase and get yourself some exposure ;)

I prefer to remain mysterious as to what I do

I just wrote it to help out the folks here.

[JFK]

Originally Posted by Varius

Well, this fell off the first page fairly quick heh...I'll give it a bump for those who may have missed it

here is another one for u

[Relentless]

This thread is so good that I'm now going back over all other posts you have made on gfy. Seriously. Thank you.

[Davy]

What software can you recommend to manage bounceback emails?

[martinsc]

great read, thanks

[DamianJ]

Originally Posted by Davy

What software can you recommend to manage bounceback emails?

Every mailing script I have used manages it for you. You can set it so that you delete after one hard bounce, 3, etc.

[Davy]

Originally Posted by DamianJ

Every mailing script I have used manages it for you.

Well, that doesn't really answer my question, does it?

[Zester]

my advice: if you are a sponsor and want your newsletter to get to your affiliate: use RSS feeds
instead of a newsletter

that is what RSS feeds are for

[DonX]

Thanks Varius, I've been looking for something like this

[Varius]

Originally Posted by Davy

What software can you recommend to manage bounceback emails?

Personally, I always opt for custom code (for example with Qmail I prefer to pipe bounces directly into a Python script which can handle them).

Having a simple PHP cronjob "pickup" script shouldn't be too hard for anyone to write.

Software-wise, I'm not too up-to-date on them but a long time ago used one called BoogiePOP which appears to still be around.

[uno]

Very well written and informative article keith!

[Brujah]

Excellent information but I'm so clueless when it comes to properly configuring all this mail stuff on my servers. Where's a good place to learn?

[Varius]

Originally Posted by Brujah

Excellent information but I'm so clueless when it comes to properly configuring all this mail stuff on my servers. Where's a good place to learn?

Honestly, since much of it varies so much depending on your MTA (mail transport agent) I haven't come across one really good all-in-one spot. For example if you run Postfix and want to install DKIM it'll be different than Qmail and DKIM. Google is your best bet.

However, feel free to ask me any specific questions and I can point you in the right direction

[wdsguy]

Which emailing software if any do you recommend we setup on our server if we are managing a small email list and we weren't going through an outside service such as aweber?

[Varius]

Originally Posted by wdsguy

Which emailing software if any do you recommend we setup on our server if we are managing a small email list and we weren't going through an outside service such as aweber?

Again, my first answer is always go custom, but here are some options for small mailing list management I would recommend:

Qmail + Qmailadmin

Qmail + EZMLM

Postfix + MySQL/PostgreSQL

As far as 3rd-party softwares the ones I have used (mostly PHP/MySQL ones out there ranging in price up to few hundred dollars) have been full of bugs and usually their support was very weak, so I don't recommend any.

Generally, every server I have ever configured I have used either Qmail or Postfix on, so most of my experience is with these two softwares. I do not recommend using the OS default Sendmail. If you are in a Windows environment, though, I'm not the right guy to offer advice as I stay away from Windows as far as mail servers go.

[wdsguy]

Originally Posted by Varius

Again, my first answer is always go custom, but here are some options for small mailing list management I would recommend:

Qmail + Qmailadmin

Qmail + EZMLM

Postfix + MySQL/PostgreSQL

As far as 3rd-party softwares the ones I have used (mostly PHP/MySQL ones out there ranging in price up to few hundred dollars) have been full of bugs and usually their support was very weak, so I don't recommend any.

Generally, every server I have ever configured I have used either Qmail or Postfix on, so most of my experience is with these two softwares. I do not recommend using the OS default Sendmail. If you are in a Windows environment, though, I'm not the right guy to offer advice as I stay away from Windows as far as mail servers go.

Much appreciated, i learned most of the things you listed in the thread the hard way. This will save people alot of time with their learning curve.

[wdsguy]

Keith, any advice for dealing with hotmail? I been trying to contact them to get whitelisted. They never seem to respond and I am not up for paying for senderscore.

[Varius]

Originally Posted by wdsguy

Keith, any advice for dealing with hotmail? I been trying to contact them to get whitelisted. They never seem to respond and I am not up for paying for senderscore.

Hotmail is the worst. There are days, for no apparent reason, they will simply junk your mail out of nowhere. Then a day or two later you will be back in.

Unless you have a contact there you are out of luck, unfortunately. You might try to contact Taryn and Michael at InboxEverytime and see if they still have a Hotmail contact; they have helped me greatly with their contacts in the past.

[Davy]

Originally Posted by Zester

my advice: if you are a sponsor and want your newsletter to get to your affiliate: use RSS feeds
instead of a newsletter

that is what RSS feeds are for

There's still a lot of people who have no clue about RSS feeds. But everybody knows email.
RSS feeds are also not so convenient, because you need to check them in a separate application.
I'd say you should have both RSS and email newsletters.

[Davy]

I wish vbulletin had an automatic bounceback email removal feature.
I'm getting swamped with thousands of messages a day.
There's one addon that claims it can do this, but I haven't looked into it, yet.

[TrainWreckContent]

thats some great info in there thanks for the read!!

[NinjaSteve]

Thanks for the article. A lot of people completely forget about email marketing.

[Naechy]

thank you for this

[czarina]

very good information. Thanks for sharing!