Amazon SES vs Mailgun - the ultimate guide

Quick Overview

Outline:

• Multiple domain and account relationship
• Pricing
• Sending
• Receiving
• Event tracking
• Mailing lists
• Analytics
• Conclusion

1. Multiple domain and account relationship

Both ESPs allow you have multiple domains within a single account. But the way the multiple domains are managed differ. This difference is important because it changes everything.

In Mailgun, each domain is treated as a separate entity. Everything related to the domain is separated from the other domains–sending keys, webhooks, mailing lists, templates and more. What you change for a domain doesn't affect the other. If a domain exceeds the bounce rate or complaint limit, it affects just that domain. The one thing Mailgun missed though is allowing team members be directly assigned to a domain. At the moment, you add people to the account.

In Amazon SES, it is the other way. Most of the settings that should be tied to the domains are tied to the account–account keys, mailing lists, configuration sets, templates and more. Bounce rates or complaints from a single domain affects the whole account and can cause the whole account to be locked.

2. Pricing

Amazon SES offers the most competitive pricing for transactional emails. No email service provider (ESP) comes close. For outbound mails, Amazon SES costs around $1 per 10k emails. That's ridiculously cheap. Mailgun costs about 8 times more.

3. Sending

SMTP and API

Both ESPs support SMTP and API for sending emails, as should any ESP anyway.

Rate limiting

Amazon SES rate limits the number of emails you can send per second and per 24 hours. You can find this limit on your dashboard. The exact value depends on your email volume and you can always request an increase. They allow you do an email burst for an unspecified short period of time but once you shoot over the limits for an extended time, the emails will bounce.

Mailgun doesn't specify exact send limits. This doesn't mean there is none. If you send more emails than your Mailgun account send rate, the emails are automatically queued and sending is throttled for you. What you should be careful of is the unspecified API request limits which will cause HTTP request error. But you must be doing something heavy to hit that.

Dedicated IP

Both ESPs allow your purchase a dedicated IP for better email deliverability. This costs $59/month in Mailgun and $24.95/month in Amazon SES. Both providers also support automatic warmup of the IP.

Scheduled send

Mailgun lets you schedule emails for send at your preferred time. You can also allow Mailgun decide the best time to send the email based on analysis of your recipients' active time. Amazon SES doesn't offer any of this. To do scheduled send with Amazon SES, an idea will be to use a delayed Amazon SQS queue that will trigger the send through Lambda.

Templates

Both ESPs allow you create email templates. Instead of sending an email with the full content, you can use a template and replace certain placeholder strings (generally called tags) in the template with actual data. One big advantage of this is that it makes managing email content a lot easier.

Consider a template with the content:

Hello {{firstname}}
Here is the link to activate your account: awesome.tld/verify/{{id}}

Sending an email with the template and the data: {"firstname": "Opeyemi", "id": 1} becomes:

Hello Opeyemi
Here is the link to activate your account: awesome.tld/verify/1

Sending an email with same template and the data: {"firstname": "Lizzy", "id": 2} becomes:

Hello Lizzy
Here is the link to activate your account: awesome.tld/verify/2

For Mailgun, the only template part is the email body. This means you still need to specify a subject when you want to send a template based email. For Amazon SES, a template comprises both the email subject and body. The advantage of this is that you do not need to specify the subject when you need to send a template based email. The disadvantage is that you cannot use the same template for an email with same content but different subjects.

You can create a Mailgun template from the dashboard. You can only create a template for Amazon SES through the API.

(By the way, you can manage your Mailgun and Amazon SES templates directly on Engage).

Interestingly, both ESPs use Handlebars as template language. This means you can perform logic actions like conditional statements and loops within your templates.

Tags

When you send an outbound email, you can tag it. Tags are useful for adding additional information or metadata not visible to the recipient to an email. They are also useful for grouping emails together for analytics purposes.

Tags in Amazon SES are in the form of "key" to "value". You assign a name to the tag and assign a value to the name. For example, you can tag an email campaign and set the value to 1234.

tags: [
  { "campaign": "1234" }
]

Tags on Mailgun work differently. A tag can only be a value. For our campaign example, it's either we tag the mail campaign or 1234. Or something like campaign-1234.

tags: [
  "campaign-1234"
]

Suppression list

Both ESPs maintain a suppression list of email addresses that caused a complaint, bounced, or unsubscribed from a campaign. Emails sent to an email address in the suppression list is automatically dropped. Both providers also allow you remove an email from the list programmatically and from the dashboard.

Testing

When you create a Mailgun account, you get a sandbox domain in the form sandbox***.mailgun.org. You can use this sandbox domain to send test emails to a few verified recipients before adding and verifying your domain. There is a second option called test mode that also works once your domain is verified. If sending through the API, you can set o:testmode to true. If sending through SMTP, add a X-Mailgun-Drop-Message header and set it to yes. Mailgun will not deliver emails sent in test mode but you will be charged for them.

In AWS SES, testing works differently. SES provides a series of emails you can send tests to depending on the wanted outcome.

• For successful delivery: [email protected]
• For bounce: [email protected]
• For automatic responses: [email protected]
• For complaints: [email protected]
• For hard bounce as if added to suppression list: [email protected]

If you need to send to multiple versions of the same test email, you can use labels e.g. [email protected]. But go easy, Amazon also charges you for test emails.

4. Receiving

Both ESPs support inbound emails. You can create inbound routes/rules which are receiving email addresses or formats that allows you process incoming emails. On Mailgun you can forward the emails to another address or a webhook hosting a custom code. Or store it or bounce. On Amazon SES, you can save it in S3 or forward it to a custom Lambda function. Or send to Amazon SNS. Or bounce.

5. Event tracking

Events tracked

Both ESPs track the following events: Delivery, Bounce (in different forms), Complaint, Click and Open. Mailgun tracks Accept and Unsubscribe in addition to these. In addition, Amazon SES tracks Sends, Delivery delay, and Rendering failures.

Click tracking

By default, Amazon SES uses its own domain for click tracking. URLs will be rewritten to a awstrack.me link. However, you can use your own custom domain for tracking.

On Mailgun, you need to setup a custom subdomain for open and click tracking. URLs are automatically rewritten to use this subdomain and for the most part, this is fine. It becomes a challenge if your site forces SSL connection. You will need to generate a certificate for that tracking domain or use a service like Cloudflare as a proxy CDN.

Ease of event tracking setup

Event tracking in Mailgun is easier than SES. The steps are:

• Create a webhook to receive the events
• Go to the webhook settings for your domain in your Mailgun dashboard and update it to your webhook.
• Your domain can have multiple webhooks listening to events

Here is a more detailed post on how to do this.

The process is more complex with AWS SES. Granted there are a couple of tracking options, if you are planning to send to a webhook, you will have to:

• Create a configuration set
• Create a SNS topic and subscription that publishes to your webhook
• Connect the SNS and configuration set
• You can have multiple configuration sets (but you can only use one at any time).

Here is a more detailed post on how to do this.

On Mailgun, events for every email you send are automatically sent to your webhooks. No need to make extra code changes to your existing send scripts. For AWS SES, you need to make slight changes to how you send your emails to use a configuration set for event tracking. You can only use a single configuration set when you send an email. This means that if you want to send the events to multiple webhooks, you have to create a configuration set with multiple event destinations.

On Mailgun, opens and clicks tracking can be customised at the point of send irrespective of the default domain settings. If sending via SMTP, you can pass a X-Mailgun-Track-Clicks or X-Mailgun-Track-Opens header to enable or disable click/open tracking. If sending through API, you can pass a o:tracking-clicks or o:tracking-opens parameter. It’s a little tricky with Amazon SES. Click tracking is the only thing you can disable at the point of send. To do this, you need to add a ses-no-track parameter to the link’s href like this: <a href="" ses:no-track>Click here</a>. I can think of an instance where this is useful though–if you want to track some links within the mail and disable some others. But as you should also already notice, disabling click tracking this way can only work for HTML emails since it requires modifying the A tag.

Event verification

Mailgun sends a signature you can use to confirm the event is from Mailgun. Verifying the signature of SES events is a little more complex.

Mail body access

There is no way to access the mail body of an email sent through Amazon SES. The most you can get from the event is the email subject. With Mailgun, a delivery event will come with a URL you can make an authenticated request to to access the email content.

If you are using Engage to manage your transactional emails through Amazon SES, you will be provided an email you can BCC your outgoing transactional emails with so you can view the email body on your Engage dashboard. This helps you see what emails your users are getting and be sure they are delivered as should be.

6. Mailing lists

Mailing lists lets you group related subscribers together so you can send a broadcast to them at once. Both ESPs allow you create a mailing list but they work in different ways.

Amazon SES's mailing list is similar to a newsletter list. You can create a list (one per account) with multiple topics (20 max). You can then subscribe contacts to the list or topic. Think of the list as your site and the topics as different sublists you subscribe contacts to .e.g Blog updates, Important Updates, Weekly updates. Contacts can belong to multiple topics. To send a mail to the list, you specify the list and topic when making a send API or SMTP request. List management can only be done through the SES API.

Mailgun's mailing list is what you will call a group list. This means subscribers can reply to the email and replies can be sent to either all other subscribers or the sender. This is how mailing lists for sites like Google groups work. To create a list, you create the list's email and decide how you want the list to work–who can send mails, if subscribers can reply and if replies should only be sent to the sender or everyone. You can manage the list from your Mailgun account dashboard as well as the API.

7. Analytics

Amazon SES has an almost non existent analytics dashboard. You can see your send, bounce and complaint rates, but you can't see the email activities. You need to use event tracking to get that data yourself.

Mailgun provides email event analytics and activities log on the dashboard. However, depending on your plan, activity log data is only retained for a few days.

Conclusion

This is not a this-is-better-than-that comparison. Different projects require different needs. However, by highlighting the core differences, it is easier to pick what would work best for a particular project. If you are doing a lot of sends, pricing will be a major factor, and it may make sense to go for Amazon SES. If for example, you are building a forum and you want a group list, then selecting Mailgun will be a no-brainer. I should also mention that Mailgun has deliverability features like email validation and inbox placement tests.

If you are looking for an analytics and reporting dashboard for your Mailgun or Amazon SES account, sign up on Engage for free. Engage lets you see in real-time your transactional email events, track bounces and complaints and manage your email templates. Beyond that, you can use Engage to send broadcasts and create message automations.

Did we miss anything in the list? Please let us know: twitter.com/engage_so