Tutorial: Exploring Azure Event Grid with Custom Webhooks

Azure Event Grid is a managed event routing service based on the publish-subscribe protocol. It enables developers to easily connect event publishers with consumers. In August 2017, Microsoft launched Event Grid service in preview. Last week, it became generally available across 10 Azure regions.

For an overview of Azure EventGrid, refer to my article published in October 2017. Since the initial preview, Microsoft has added multiple event sources and destinations including Azure IoT Hub and Azure Event Hubs. The service now comes with 99.99 percent availability with a financially backed SLA.

Apart from consuming the events generated by Azure resources, developers can use Event Grid as a generic messaging infrastructure. This capability enables serverless applications and microservices to seamlessly communicate with each other.

One of the consumers of Event Grid messages is a custom WebHook. With this integration, it is possible to trigger events running in a variety of environments including Functions as a Service (FaaS) or custom REST endpoints running behind firewalls.

In this tutorial, we will explore a scenario where an on-premises web service gets a notification each time a file is uploaded or deleted from an Azure Storage Container. We will use this scenario to understand the schema of Azure Event Grid subscriptions. Though the REST endpoint is based on Node.js Express framework, it can be easily extended to Serverless environments such as Azure Functions or Apache OpenWhisk.

👁 Image

Before we get started with the step-by-step guide, make sure that you have an active Azure subscription. You also need to have the latest version of Azure CLI, ngrok and node.js installed on your machine.

Let’s start by setting up the environment. Open a new terminal window and start ngrok on port 8000. This will open an HTTPS tunnel to your local machine making it possible to receive messages from Azure Event Grid.

$ ngrok http 8000

👁 Image

Open another terminal window, and run the following commands to set the environment variables. Don’t forget to replace the END_POINT variable with the actual ngrok URL.

$ LOC=westcentralus
$ RG=aegdemo
$ AZURE_STORAGE_ACCOUNT=input7675
$ END_POINT=https://e7fa39a5.ngrok.io

To ensure that Microsoft.EventGrid is registered in your account, run the below commands.

$ az provider register --namespace  Microsoft.EventGrid
$ az feature register --name storageEventSubscriptions --namespace Microsoft.EventGrid
$ az feature show --name storageEventSubscriptions --namespace Microsoft.EventGrid

You should see the output similar to the following. This confirms that Azure EventGrid is available for your subscription.

{
 "id": "/subscriptions/<subscription_id>/providers/Microsoft.Features/providers/Microsoft.EventGrid/features/storageEventSubscriptions",
 "name": "Microsoft.EventGrid/storageEventSubscriptions",
 "properties": {
   "state": "Registered"
 },
 "type": "Microsoft.Features/providers/features"
}

Let’s go ahead a create an Azure Resource Group for the resources that we will create.

$ az group create --name $RG  --location $LOC
{
 "id": "/subscriptions/<subscription_id>/resourceGroups/aegdemo",
 "location": "westcentralus",
 "managedBy": null,
 "name": "aegdemo",
 "properties": {
   "provisioningState": "Succeeded"
 },
 "tags": null
}

Next, we will create an Azure Storage account for storing blobs. The JSON output confirms that the storage account is successfully created.

$ az storage account create \
 --name $AZURE_STORAGE_ACCOUNT \
 --location $LOC \
 --resource-group $RG \
 --sku Standard_LRS \
 --kind BlobStorage \
 --access-tier Hot

Before we go any further, it’s time for us to launch the Node.js program that will start listening for the requests routed via ngrok. You also need to install Express framework before executing the code.

'use strict';
var express = require('express');
var bodyParser = require('body-parser');
var webhook_res=""
var app = express();

app.use(bodyParser.json());
app.post('/',function (request, response) {
//  console.log(request.body);
 if('data' in request.body[0]){
   if('validationCode' in request.body[0].data) {
     webhook_res = {'validationResponse': request.body[0].data.validationCode}
     console.log('Azure EventGrid subscription successfully validated')
     response.send(webhook_res);
   }

     if(request.body[0].data.api=='PutBlob'){
       console.log('>> Blob uploaded - %s', request.body[0].data.url)
     }
     if(request.body[0].data.api=='DeleteBlob'){
       console.log('>> Blob deleted - %s', request.body[0].data.url)
     }
     response.send();
     response.end();
   }
});
var server = app.listen(process.env.PORT || '8000', function () {
 console.log('App listening on port %s', server.address().port);
 console.log('Press Ctrl+C to quit.');
});

Save and run the above code in a new terminal window:

$ node app.js
App listening on port 8000
Press Ctrl+C to quit.

The crux of this program lies in the POST handler that deals with the WebHook requests hitting the endpoint.

There are three types of requests that this application will handle —

1. Subscription validation
2. Blob creation
3. Blob deletion

When we register our own WebHook endpoint with Event Grid, it sends a POST request with a simple validation code in order to prove endpoint ownership. The app needs to respond by echoing back the validation code. Event Grid does not deliver events to WebHook endpoints that have not passed the validation.

To enable this validation, we check for validationCode within the body and embed the same code in the response. The following code snippet is responsible for this validation.

   if('validationCode' in request.body[0].data) {
     webhook_res = {'validationResponse': request.body[0].data.validationCode}
     console.log('Azure EventGrid subscription successfully validated')
     response.send(webhook_res);
   }

Subsequent requests generated by Azure Event Grid will not contain the validation code. It’s only required during the subscription. Once the subscription is made, we will receive requests when a new blob is uploaded or an existing blob is deleted. The code snippets shown below tackle these events.

     if(request.body[0].data.api=='PutBlob'){
       console.log('>> Blob uploaded - %s', request.body[0].data.url)
     }
     if(request.body[0].data.api=='DeleteBlob'){
       console.log('>> Blob deleted - %s', request.body[0].data.url)
     }

In our sample, we simply print the URL of the blob. This scenario can be easily extended to powerful use cases such as backing up the blobs to the local storage or to other cloud storage environments.

With the WebHook in place, it’s time for us to create an Event Grid subscription.

$ az eventgrid resource event-subscription create \
--endpoint $END_POINT  \
--name aegdemosub \
--provider-namespace Microsoft.Storage \
--resource-type storageAccounts \
--resource-group $RG \
--resource-name $AZURE_STORAGE_ACCOUNT

Notice that we are instructing Event Grid to handle events raised by storage accounts.

When you run the above command, our Node.js app responds to show the following output.

$ node app.js
App listening on port 8000
Press Ctrl+C to quit.
Azure EventGrid subscription successfully validated

You will also see the JSON payload emitted by the Azure CLI. This indicates that the subscription is successfully registered with the WebHook.

We will now create a container in Azure Storage to hold the uploaded files. To complete this step, we need to store the access key in the AZURE_STORAGE_ACCESS_KEY environment variable. The below commands handle these steps.

$ export AZURE_STORAGE_ACCOUNT=input7675
$ export AZURE_STORAGE_ACCESS_KEY="$(az storage account keys list --account-name $AZURE_STORAGE_ACCOUNT --resource-group $RG --query "[0].value" --output tsv)"
$ az storage container create --name aegdemo
{
 "created": true
}

With the storage container in place, let’s go ahead and upload a file to Azure Storage.

$ echo "Hello Azure Event Grid" > hello.txt
$ az storage blob upload --file hello.txt --container-name aegdemo --name hello.txt
Alive[################################################################]  100.000Finished[#############################################################]  100.0000%
{
 "etag": "\"0x8D568A4964D19FD\"",
 "lastModified": "2018-01-31T12:17:24+00:00"
}

Now, check the terminal window running the Node.js application. It prints the URL of the blob.

App listening on port 8000
Press Ctrl+C to quit.
Azure EventGrid subscription successfully validated
>> Blob uploaded - https://input7675.blob.core.windows.net/aegdemo/hello.txt

Let’s go ahead and delete the blob.

$ az storage blob delete --container-name aegdemo --name hello.txt
{
 "deleted": null
}

Don’t forget the to check the output of the application. You will see the response to all the three requests — subscription validation, blob creation, and blob deletion.

👁 Image

Perform clean up by running the below commands.

# Delete the blob container
$ az storage container delete -n aegdemo
# Delete AEG subscription
$ az eventgrid resource event-subscription delete \
--name aegdemosub \
--provider-namespace Microsoft.Storage \
--resource-type storageAccounts \
--resource-group $RG \
--resource-name $AZURE_STORAGE_ACCOUNT
# Delete Azure Storage account
$ az storage account delete \
--name $AZURE_STORAGE_ACCOUNT \
--resource-group $RG
# Delete Azure Resource Group
$ az group delete -n $RG

The objective of this tutorial is to walk you through all the steps involved in creating an Azure Event Grid subscription and consuming the events through a custom WebHook. I chose the CLI route to highlight the workflow involved in this setup.

In one of the upcoming tutorials, I will demonstrate how to extend this scenario to create a serverless, multi-cloud replication of Azure Storage accounts. Stay tuned!