Incident.io Webhook Integration

Trustgrid can be integrated with Incident.io using the generic webhook channel to send alerts based on Trustgrid events. This allows for streamlined incident management and improved response times. The guide below outlines the steps to set up this integration along with suggestions on gathering additional attributes for the alerts.

Setting Up the Integration

Step 1 - Start Creating the Incident.io Alert Source

1. Log in to your Incident.io account.
2. Click on your organization name at the top and then select Settings.
3. Under On Call, click on Alerts.
4. Click the + to the right of Sources to create a new source.

   

5. Follow these steps to create the source

   

   1. Search for http.
   2. Select the HTTP type from the list and provide a unique name.
   3. For Type of HTTP Source select custom. This allows the source to use the native Trustgrid JSON payload which can be transformed into the Incident.io format.
   4. Click Continue to proceed to the next step.
6. Capture the generated webhook URL and save it for later use.
   1. Select the Query Authentication tab.
   2. Click the copy icon to the right of the URL. Save this as you will need it later .
7. Set the Transform Expression and Deduplication key path
   1. Scroll down until you see the Transform expression section. There will be existing JavaScript there. Select all and delete.
   2. Copy and paste the following code into the Transform expression section:

      var resolvedFlag = !!$.resolved; // force boolean
      var statusValue = resolvedFlag ? "resolved" : "firing";
      
      return {
        title: $.nodeName + " - " + $.eventType,
        description: $.message,
        status: statusValue,
        deduplication_key: $.uid,
        metadata: {
          nodeName: $.nodeName,
          eventType: $.eventType,
          level: $.level,
          domain: $.domain,
          tags: $.tags,
          timestamp: $.timestamp
        }
      };
      

      Copy

      

   3. Scroll below to the Deduplication key path section and set the value to $.uid.
   4. Click Save configuration to finish creating the webhook source.

      

8. On the right side, you should see a section saying No alerts received yet. In the next section, you will create a Webhook channel in Trustgrid and trigger events to generate sample Alerts for Incident.io to use to complete the configuration.

Step 2 - Create Webhook Channel in Trustgrid

To complete the below step it is idea to identify a node that you can use to generate test events, ideally by performing an action such as restarting the node. In the example below we will create a new Alarm Filter for a specific node and test by restarting it.

1. Login to the Trustgrid portal.
2. Navigate to Alarms > Channels.
3. Click the +Create Channel button. (Or optionally select and choose Edit from Actions to add to an existing Channel)
4. In the channel configuration, set the following:
   1. Name: Give your channel a descriptive name.
   2. Generic Webhook: Paste the URL you copied from Incident.io.
   3. Click Save to create the channel.

      

5. Setup an Alarm Filter to test the new channel. (If you have an existing Alarm Filter that you can use for testing, just edit and select the channel created above and save. Then proceed to the next step.)
   1. Navigate to Alarms > Alarm Filters.
   2. Click the +Create Alarm button.
   3. In the filter configuration, set the following:
      1. Name: Give your filter a descriptive name. This example is just for testing.
      2. Make sure Enabled is checked.
      3. Node name: Select the node you identified for testing.
      4. Severity Threshold: Change to INFO so that all relevant events match the filter including Resolved notifications.
      5. Channels: Select the channel you created above.
      6. Click Save to create the filter.

         

6. To test the filter, restart the node you selected in the filter. This will generate a Node Restart event, then a Node Disconnect event followed by a Node Connect event.

The Trustgrid system will only send out matching alerts if there are no unresolved events of that type. If the test node has unresolved events, use the Alert Center to clear the:

1. Click the Alert Center icon in the top right corner of the Node’s detail page.
2. You can use the check marks to resolve individual Alerts, or
3. Click the Mark All As Resolved button to clear all unresolved Alerts of that type.

Step 3 - Complete Alert Source config in Incident.io

To complete the configuration we will use the generated Alerts to extract additional useful attributes from the example events to surface in Incident.io. Feel free to modify the Attribute names to fit your needs.

1. Return to the Alert sources setup page in Incident.io
   1. You should now see example alert in the bottom right.
   2. Click Continue.

      

2. On the Configure your setup page, you can map the attributes from the incoming alerts to the fields in Incident.io. Use the example alert to help you identify the relevant fields.
   1. Leave the default Alert title and Description mappings unless you need to customize them.
   2. Click the Edit pencil icon to the right of Attributes.

      

      1. In the Alert Payload section, click +“nodeName”. As the name implies, this will include the name of the Node the event relates to.

         

         1. Unless you have an existing relevant attribute defined, such as Server Name, you can create a new one. Scroll down and select + Add new attribute.

            

         2. In the new attribute configuration:
            1. Give the attribute something descriptive, like Node Name. Leave all other settings at their defaults.
            2. Click Add

               

      2. Repeat the above steps to add the following attributes:
         1. +eventType - This will include the type of event that triggered the alert.
         2. +description - This will copy the Description, which is the original Event’s message, into an attribute. This will allows this to be displayed when the Alert is forwarded to a tool like Slack without having to open the actual alert.
         3. +level - This will include the severity level of the event.
         4. +domain - This is only useful if your company has multiple Trustgrid organizations with different domains. This will allow you to potentially filter and route differently based on this attribute.
         5. +tags - This will include all configured [Tags] /docs/nodes/shared/tags/. For this attribute, change the Choose what results should be parsed into to Labels.

            

   3. Click Apply in the bottom right to complete adding Attributes.
3. Click Save and finish to complete the Alert configuration in Incident.io
4. At this point you can follow the Incident.io documentation to Create a new Alert route from the new alert source.

Addition Information

Understanding the transformations

 1  var resolvedFlag = !!$.resolved; // force boolean
 2  var statusValue = resolvedFlag ? "resolved" : "firing";
 3  
 4  return {
 5    title: $.nodeName + " - " + $.eventType,
 6    description: $.message,
 7    status: statusValue,
 8    deduplication_key: $.uid,
 9    metadata: {
10      nodeName: $.nodeName,
11      eventType: $.eventType,
12      level: $.level,
13      domain: $.domain,
14      tags: $.tags,
15      timestamp: $.timestamp
16    }
17  };

• Lines 1-2 are responsible for determining the alert’s status based on its resolution state. This is then convert into Incident.io’s syntax of firing for new alerts and resolved for resolved alerts.
• Lines 4-20 define the structure of the alert object that will be sent to Incident.io.
• Line 5 sets the title of the alert using the node name and event type.
• Line 6 sets the description of the alert using the message from the original event.
• Line 7 sets the status of the alert using the resolved flag to determine if it is firing or resolved.
• Line 8 sets the deduplication key for the alert using the unique ID from the original event. This allows Incident.io to match a firing alert with its corresponding resolved alert.
• Lines 9-20 set various metadata fields for the alert, including the node name, event type, level, domain, tags, and timestamp.

Testing via Curl

If you want to test the webhook integration, you can use Curl to send a sample payload to the Incident.io webhook URL.

This can be useful if you are not seeing the expected alerts in Incident.io after triggering events in the Trustgrid portal as the curl command may return any HTTP error Incident.io is responding with.

Here’s an example command:

curl -X POST <WEBHOOK_URL> \
-H "Content-Type: application/json" \
-d '{
	"nodeName": "edge1", 
	"expires": 1604801325, 
	"level": "INFO", 
	"eventType": "Test Event",
	"source": "EKG",
	"message": "This is just a test event. It is not real.",
	"type": "Alert",
	"orgId": "8e1c2c05-2c86-4b1b-a0cc-############",
	"GS1PK": "Org#8e1c2c05-2c86-4b1b-a0cc-############",
	"_ct": {},
	"uid": "1jwV1R2R6itQUjPza9yqTE8a8zu",
	"GS1SK": "Alert#1jwV1R2R6itQUjPza9yqTE8a8zu",
	"_md": {},
	"domain": "example.trustgrid.io",
	"SK": "Alert#Node Disconnect",
	"_tp": "Alert",
	"PK": "Node#0895b104-5434-447b-8577-############",
	"state": "UNKNOWN",
	"nodeId": "0895b104-5434-447b-8577-############",
	"timestamp": 1604714923, 
	"channelID": "bc47ca84-1d04-454b-bedc-a55d1a917c0e", 
	"notes": ["Text from Description Field"],
    "tags": { "prod_status":"production","site_name":"Main Datacenter"},
    "resolved": false
}

Replace <WEBHOOK_URL> with the actual URL of your Incident.io webhook. This command will simulate an alert being sent to Incident.io, allowing you to verify that the integration is working correctly.

If everything is working you should see a message like {"status":"accepted","message":"Event accepted for processing","deduplication_key":"735cbacc3f07e740d26ff364a19f856aa5af95f929017538214093afb132006e"} and then the Alert will show in Incident.io.