Getting email event data with Mailgun webhooks

Tracking email events like opens, clicks, failures and complaints can be done on Mailgun using Mailgun webhooks. Although this data is available in the Mailgun account dashboard, there are many reasons using the webhooks may come in handy. Because the webhook is a realtime push to you, you can collect the information to do things like visualisation, reporting or presentation of the mail data in any form or through any channel you want. There is also an API to get the events but using the webhooks has more advantages.

1. You don’t have to make repeated API requests. This means lesser resources.
2. Updates are more realtime because it is pushed as soon as available.

Outline:

• Setting it up
• Handling the data
• Making it secure
• Making it scalable
• Conclusion

Setting it up

There are two ways to set up webhooks in Mailgun. It can either be done through the Mailgun dashboard or API. The more straightforward way to do it is through the dashboard. This is available through Sending -> Webhooks on the Mailgun dashboard.

Handling the data

To handle the event data sent to our webhook, we must know what the data looks like in the first place. The API documentation shows this but we can confirm by using a test webhook that will log the data from Mailgun. We can use Mailgun’s Postbin or Webhook.site to do this. These services will generate a unique endpoint we can use as our webhook. We will be using Webhook.site for this post.

Let’s go ahead to create a test endpoint and see what the event data look like compared to what is in the documentation.

• Visit webhook.site
• Copy the generated URL and access the Webhooks section of your Mailgun dashboard.
• Paste the URL in the "URL to test" input field of the Test webhook section and click the Test Webhook button. This will send sample event data to the URL.
• Repeat this for all the events you are interested in. Ignore the ones labeled Legacy.

The event data will become automatically available in webhooks.site and you can review the data for the different events. With that, it is easy to write the code that will handle the sent data. Here is a simple code that will output details of complaints and dropped emails.

const express = require(‘express’)
    , bodyParser = require(‘body-parser’)
    , multer = require(‘multer’)
    ;

const app = express();
app.use(bodyParser.urlencoded({extended: false}));
app.listen(process.env.PORT || 3000);

app.post(‘/webhook’, multer().none(), function(req, res) {
  const email = req.body.recipient;
  const event = req.body.event;

  if (event == ‘complained’) {
    console.log(`${email} complained about your mail`);
  }
  else if (event == ‘dropped’) {
    console.log(`Mail to ${email} dropped. ${event.description}`);
  }
  else if (event == ‘bounced’) {
    console.log(`Error ${event.code}: Mail to ${email} bounced. ${event.error}`);
  }

  res.end();
});

Making it secure

Nothing stops anyone that knows our webhook from crafting a false event data and sending it to the URL. Luckily, Mailgun signs each request sent and posts the following parameters along:

• timestamp (Number of seconds passed since January 1, 1970)
• token (Randomly generated string with length 50)
• signature (Hexadecimal string generated by HMAC algorithm)

To verify the token:

• Concatenate the values of timestamp and token.
• Encode the resulting string with HMAC, using your Mailgun HTTP signing key (available in your Mailgun dashboard) as the key and Sha256 as the algorithm.
• The result should be the same as the signature.

Here is what it looks like in Node.js:

const value = event_data_timestamp+event_data_token;
const hash = crypto.createHmac(‘sha256’, apikey)
                  .update(value)
                  .digest(‘hex’);
if (hash !== event_data_signature) {
  console.log(‘Invalid signature’);
  return;
}

If we add that to our original code example, it becomes this:

const express = require(‘express’)
    , crypto = require(‘crypto’)
    , multer = require(‘multer’)
    , bodyParser = require(‘body-parser’)
    ;

const app = express();
app.use(bodyParser.urlencoded({extended: false}));
app.listen(process.env.PORT || 3000);

app.get(‘/webhook’, multer().none(), function(req, res) {
  // Validate signature
  const value = req.body.timestamp+req.body.token;
  const hash = crypto.createHmac(‘sha256’,
              process.env.API_KEY)
                  .update(value)
                  .digest(‘hex’);
  if (hash !== req.body.signature) {
    console.log(‘Invalid signature’);
    return res.end();
  }

  // Log status of event
  const email = req.body.recipient;
  const event = req.body.event;

  if (event == ‘complained’) {
    console.log(`${email} complained about your mail`);
  }
  else if (event == ‘dropped’) {
    console.log(`Mail to ${email} dropped. ${event.description}`);
  }
  else if (event == ‘bounced’) {
    console.log(`Error ${event.code}: Mail to ${email} bounced. ${event.error}`);
  }

  res.end();
});

We can even step this up and:

1. For every request, check against a token cache to prevent use of the same token. Every token will be stored there. This will prevent replay attacks.
2. Check if the timestamp is not too far from the current time.

Making it scalable

If you send lots of emails and you are expecting lots of events, putting your webhook code on a server that can’t scale automatically is a bad idea. Even if you are not expecting lots of events, unexpected things can lead to a surge in events. Having a server that can scale automatically is really useful for instances like this.

Enter Serverless computing. The idea of serverless computing is that you can delegate the execution of your code and everything related to another service and not worry about server management. Because multiple instances of your code can be executed in parallel and you can adjust computing resources like RAM and execution time on the fly, it is highly scalable. You are also charged based on consumed resources and execution time so it can be really cheap.

There are a couple of serverless computing providers but for this post, we will use Google Cloud Functions because of the ease of setting up HTTP functions. An HTTP function is a code block wrapped as a function that can be triggered by visiting a URL. This is exactly what we need as our webhook.

To create this function, we need to write a JavaScript function that will be exported as a Node.js module. The function takes HTTP-specific arguments: `request` and `response`.

exports.webhook = function(request, response) {
  // Handle event data here
  response.send({status:”ok”});
}

Depending on the request content-type, the body of the request is automatically parsed and available in the body parameter of the request object.

exports.webhook = function(request, response) {
  let event = request.body.event; // delivered
  // Handle event data here
  // …
  response.send({status:”ok”});
}

Next, we publish the function to Cloud Functions. An easy way to do this is to do it from the Cloud Functions dashboard.

• Go to your Google Cloud Console (if you don’t have an account yet, create one).
• Enable Cloud Functions in the dashboard.
• Click on Create Function.
• Enter a name for your function (e.g mailgun-webhook).
• In the trigger section, select HTTP trigger. Note the URL, that will be your webhook.
• Copy your event data handling code to the index.js section of the Cloud function.
• Copy the content of your package.json and paste in the package.json section.
• Select or create a Stage bucket. The stage bucket is where the code is staged. You can use anything here.
• In Function to execute, enter the name of your function (e.g webhook).
• Save.

You can now use the function’s URL in Mailgun as your webhook.

Conclusion

There are many ways the event data from your Mailgun webhook can be used to enrich your applications outside Mailgun. If for example you allow your users send emails from your application and you use Mailgun, you can use this to provide analytics for them. Or maybe you want to send your email analytics to another platform. Or maybe you want to be notified of failures in your Slack account. Or maybe not even that. Maybe you just want more detailed analytics than what is available on the Mailgun dashboard. Whatever the use case is, getting a live data stream of mail events can really be helpful.

If you are interested in a more detailed analytics, reporting and intelligence for your transactional emails through Mailgun, without setting all these yourself, create a free Engage account.