feature image for incident reporting

Incident Reporting with PagerDuty and Nexmo

Published August 21, 2019 by Dotun Jolaoso

blog spotlight banner

Introduction

PagerDuty is an incident reporting management system that provides notifications and automatic escalations to help engineering teams detect and fix issues as they arise with their infrastructure. In this post, we’ll be looking at how we can use Nexmo to alert members of an engineering team whenever an incident occurs on any of their infrastructure.

Using Nexmo’s Dispatch API, which provides an automatic failover option, we’ll first send out notifications via Facebook Messenger as the primary medium and then SMS as a fallback option.

Technical Requirements

To follow along, you’ll need the following:

  • PHP version 7.1 or Higher
  • Laravel 5.8
  • Ngrok which allows you to expose your local web server to the internet. To learn more about how to set up your local environment with Ngrok, you can check out the docs here.
  • Composer
  • A Nexmo Account
  • Nexmo CLI
  • A Facebook Account
  • A PagerDuty Account

Setting up Nexmo

To be able to use the Dispatch API, there are a few things we need to do with our Nexmo account. Head over to your Nexmo dashboard and under the Messages and Dispatch section, create a new messages application.

new message app in dashboard
* The Status URL is the webhook endpoint Nexmo will make a POST request to whenever we send out SMS notifications. This will allow us to determine the status update of the SMS that has been sent out i.e delivered, rejected, or submitted. Since we’ll be using Ngrok, the status URL should be something like this qc43v7.ngrok.io/webhooks/status.
* The Inbound URL is the endpoint Nexmo will make a request to whenever inbound messages are received and should look similar to this qc43v7.ngrok.io/webhooks/inbound-message
* Next, generate a public/private key pair. This will automatically download the private key file for you as well. Take note of this file as we’ll be needing it shortly.
* Create the application and then take note of the application id. Next, you’ll be prompted to associate numbers and external accounts to this application. This is totally optional, so you can go just go with the default and then create the application.

Setting up PagerDuty

Head over to PagerDuty and create an account if you don’t already have one. We’ll also need to create a Service. A Service in PagerDuty represents anything we would like to open incidents against. It could be an application, a component, or even a team. So whenever a new incident occurs on that Service, we want to send out webhook notifications to a particular endpoint in our application.

In your PagerDuty dashboard, head over to Configuration -> Services and create a new Service.

pager duty dashboard
Since we’ll be making use of our own custom integration, select use our API directly for the integration type, and then select Events API V2 and fill out all the other necessary details. You can read more about creating a service on PagerDuty here.

Next, after you’ve successfully created the service, under the Integrations tab, select New Extension and then give it a type of Generic V2 webhook. The details section will consist of the endpoint in our application where we want to receive notifications from PagerDuty and should be something like qc43v7.ngrok.io/webhooks/incident

pager duty dashboard

Routes and Controller

Edit the routes/web.php with the following code:

Cross-site request forgery attack is a type of malicious attack whereby unauthorized commands are carried out on behalf of an authenticated user. To prevent such attacks, Laravel automatically generates a Csrf token for every authenticated user which helps to validate that the authenticated user is the one actually making a request to the application. However, since these requests will be coming from outside the application and we trust the source, we need to disable Csrf validation for these routes. The VerifyCsrfToken middleware is used for validating all tokens. Luckily, the middleware accepts an except array which contains a list of all routes to disable CSRF verification for.

Edit the app\Http\Middleware\VerifyCsrfToken.php file with the following code:

Next, run the following command to create a controller:

This will generate a WebhookController.php file for us in the app\Http\Controllers directory. Edit the file with the following code:

For the inboundMessage() and status() methods, we simply log the request data from Nexmo to the log file Laravel provides. The report() method dispatches a ReportIncident job which we shall be creating shortly. This job is responsible for triggering SMS notifications using a custom Wrapper around Nexmo’s API.

Linking Facebook With Your Nexmo Account

To be able to successfully send messages from Nexmo to Facebook, you’ll first need to link a Facebook page associated with your Facebook account to your Nexmo account. You can learn how to do that here.

When you’re done with the linking, send a message from your Facebook account to the Facebook page. Since it’s an inbound message, Nexmo will send a request containing details of the message to the /webhooks/inbound-message endpoint we created earlier. Check your logs file and you should see an entry similar to the following:

The Facebook sender id is the to.id, while the recipient id is the from.id. Take note of these details as we’ll be needing them shortly.

Creating a Custom Wrapper

Since the Dispatch API is still in beta as at the time of writing this tutorial, the Nexmo PHP library doesn’t yet provide support for it. As a result, we’ll be using a custom wrapper to interact with Nexmo’s API. To get started, we first need to generate a Json Web Token (JWT) which we’ll use for authenticating with the API.

Using Nexmo’s CLI, run the following command:

./private.key is the path to the private key file that was generated for you when you created the Messages and Dispatch application, while the application_id is the application id we took note of earlier. Next, copy the output of this command.

Environment and Config Variables

Add the JWT you just generated to your .env, the messenger details we noted earlier and the phone numbers that will handle sending and receiving SMS notifications.

Next, we will reference the environment variables we just defined through our config file. Head over to the config directory and under the services.php file, add a new Nexmo array.

Installing Dependency

The only dependency our project will have is the GuzzleHTTP library which we’ll use for making API calls. Run the following command to install the library:

Creating the Wrapper

In the app directory, create a Nexmo.php file and add the following code to the file:

In the dispatch() method, we’ve defined a failover template and a workflow. Whenever an incident occurs via PagerDuty, we’ll get a notification about the incident on Facebook Messenger. If the notification isn’t read within 15 seconds, the failover condition is triggered and an SMS notification is sent as a fallback.

Creating the Job

Run the following command on the terminal to create a new job class:

Edit the app\Http\Jobs\ReportIncident file with the following code:

The handle() method receives the custom Nexmo wrapper we created earlier on as a dependency and then we call the dispatch method on the class.

Bringing It All Together

It’s time to finally test what we’ve been building so far. Head over to your PagerDuty dashboard and create a new incident.
incidents screenshot

This will trigger a webhook notification to be sent out to the endpoint we added earlier to our PagerDuty account which will, in turn, send out a message to the Facebook page we linked earlier to our Nexmo account. Assuming everything went fine, you should inspect your log file as Nexmo would have sent an update as regards the status of the message we just submitted.

From the above logs, we first get a status update informing us that the message has been read and then subsequently another status update informing us that the dispatch workflow has been completed. Since the message was read within 15 seconds, the failover condition was satisfied; as a result, the SMS notification was never triggered.

To test that the SMS notification will be sent out if the failover condition isn’t met, You can repeat the same process but this time do not read the Facebook messenger notification. You’ll discover that this time the SMS notification will be triggered.

Conclusion

In this tutorial, we’ve seen how we can build a custom integration with PagerDuty and Nexmo to ensure a resilient fall back alert system in the case of an emergency. You can find a link to the Github repo here.

Leave a Reply

Your email address will not be published.