Trying to register a webhook with PagerDuty to get updates when PagerDuty updates a linked incident. Pager Duty correctly registers the serverless app URL and returns a payload with the registration information when the serverless app is configured. The problem is that the Content-Type header in the PagerDuty response is application/vnd.pagerduty+json; charset=UTF-8 which Freshworks rejects. The content type is not changeable so if Freshworks does not accept it and does not return the 200 status from PagerDuty how can my serveless app get the WebHook registration and know it is correctly installed?
Hi @IanAber, I tested with a platform v2.3 app with request templates. I added these headers:
"headers": {
"Content-Type": "application/json",
"Authorization": "Token token=<%= iparam.apiKey %>",
"Accept": "application/vnd.pagerduty+json;version=2"
}
If you are using an app on platform v2.2, can you try adding the same headers to your request method call?
Here is the full template for platform 2.3 (in config/requests.json):
{
"listPagerDutyIncidents": {
"schema": {
"method": "GET",
"host": "api.pagerduty.com",
"path": "/incidents",
"query": {
"date_range": "all",
"time_zone": "UTC"
},
"headers": {
"Content-Type": "application/json",
"Authorization": "Token token=<%= iparam.apiKey %>",
"Accept": "application/vnd.pagerduty+json;version=2"
}
}
}
}
Edit: I took these headers from PagerDuty’s Postman collection.
1 Like
This is the call I am making…
onAppInstallHandler: function () {
console.log(“Registering”);
generateTargetUrl().done(function (targetUrl) {
let bodyJSON = {
webhook_subscription: {
type: “webhook_subscription”,
active: true,
delivery_method: {
temporarily_disabled: false,
type: “http_delivery_method”,
url: targetUrl,
custom_headers:
},
description: “Updates to FreshService”,
events: [
“incident.acknowledged”,
“incident.annotated”,
“incident.delegated”,
“incident.escalated”,
“incident.priority_updated”,
“incident.reassigned”,
“incident.reopened”,
“incident.resolved”,
“incident.responder.added”,
“incident.responder.replied”,
“incident.triggered”,
“incident.unacknowledged”
],
“filter”: {
“type”: “account_reference”
}
}
};
$request.invokeTemplate(“RegisterPagerDutyWebhook”, {
body: JSON.stringify( bodyJSON )
}).then((response) => {
console.info(“Webhook registration succeeded.”);
console.info(response);
renderData();
}, (error) => {
console.error(‘Webhook Registration failed - Response from PagerDuty’);
console.error(error);
renderData({message: error.status});
});
}),
(error => {
console.error(‘Webhook Registration failed - Failed to ge the URL’);
console.error(error);
renderData({message: error.status});
});
},
This is the template…
“RegisterPagerDutyWebhook”: {
“schema”: {
“protocol”:“https”,
“method”: “POST”,
“host”: “api.pagerduty.com”,
“path”: “/webhook_subscriptions”,
“query”: {
“date_range”: “all”,
“time_zone”: “UTC”
},
“headers”: {
“Content-Type”: “application/json”,
“Authorization”: “Token token=<%= iparam.pager_duty_api_key %>”,
“Accept”: “application/vnd.pagerduty+json;version=2”
}
}
},
This is the response…
The response form Freshworks is that an error occurred and that if it repeats that I should contact support.
This is the call I am making…
onAppInstallHandler: function () {
console.log(“Registering”);
generateTargetUrl().done(function (targetUrl) {
let bodyJSON = {
webhook_subscription: {
type: “webhook_subscription”,
active: true,
delivery_method: {
temporarily_disabled: false,
type: “http_delivery_method”,
url: targetUrl,
custom_headers:
},
description: “Updates to FreshService”,
events: [
“incident.acknowledged”,
“incident.annotated”,
“incident.delegated”,
“incident.escalated”,
“incident.priority_updated”,
“incident.reassigned”,
“incident.reopened”,
“incident.resolved”,
“incident.responder.added”,
“incident.responder.replied”,
“incident.triggered”,
“incident.unacknowledged”
],
“filter”: {
“type”: “account_reference”
}
}
};
$request.invokeTemplate(“RegisterPagerDutyWebhook”, {
body: JSON.stringify( bodyJSON )
}).then((response) => {
console.info(“Webhook registration succeeded.”);
console.info(response);
renderData();
}, (error) => {
console.error(‘Webhook Registration failed - Response from PagerDuty’);
console.error(error);
renderData({message: error.status});
});
}),
(error => {
console.error(‘Webhook Registration failed - Failed to ge the URL’);
console.error(error);
renderData({message: error.status});
});
},
This is the template…
“RegisterPagerDutyWebhook”: {
“schema”: {
“protocol”:“https”,
“method”: “POST”,
“host”: “api.pagerduty.com”,
“path”: “/webhook_subscriptions”,
“query”: {
“date_range”: “all”,
“time_zone”: “UTC”
},
“headers”: {
“Content-Type”: “application/json”,
“Authorization”: “Token token=<%= iparam.pager_duty_api_key %>”,
“Accept”: “application/vnd.pagerduty+json;version=2”
}
}
},
This is the response…
An error has occurred
Hi @IanAber,
Thanks for the update. Please remember to change the API key you have posted here because this is publicly visible.
Can you clarify if:
- You are getting the error in the
$request.invokeTemplate()call fromonAppInstallHandler(). If so, can you share the error log? - You have a corresponding handler set up for
onExternalEventto listen to incoming webhook requests.
One of the platform’s limitations today is that the request proxy, and all HTTP communications in general, officially supports only the application/json and application/xml content types. With platform v2.3, I tested sending GET requests to the PagerDuty API and found that they are going through. It could be that these restrictions still apply to webhook registrations. I have asked our engineering team for clarification on this. I will keep you posted once I hear back from them.
Sorry about the delay here, @IanAber.
The developer platform supports limited content-types for incoming webhook requests (external events). They include: application/json, application/xml, text/html, text/xml, application/jsonp, text/plain, text/javascript, application/vnd.api+json.
Similar restrictions apply for sending requests using the request method.
Given that I was able to use the Content-Type: application/json with the custom Accept header to GET a successful response from the Pagerduty API in a custom app using platform v2.3, I wanted to check if your experience is different.
Here is the example app I used to test: pagerduty-test2.zip (4.5 KB).
That said, the main two paths of investigation here are:
- If you see unsupported content-type error when sending a request to the Pagerduty API using
$request.invokeTemplate(), our team look into the request method. - If you are able to register a webhook, but are not able to receive incoming webhooks after successful registration (where
onExternalEventhandler is not triggered), our team will look into external events.
If you use Postman to send your request, the GET incidents request that you gave as an example, you will see that the payload returned has a header that indicates a content type of application/json. This works using the request object as you would expect.
If you use the same request mechanism but send a POST request for the webhook-subcriptions endpoint the response from Pager Duty contains a content-type header of application/vnd.pagerduty+json; charset=UTF-8. This fails when trying to use the request object.
The failure is on the receipt of the registration payload from PagerDuty. The actual command has been processed as expected by Pager Duty without error but when pager duty sends back the webhook ID it is lost because Freshworks rejects the packet based on content type.
To get this to work in a development setting, i.e. running the serverless app locally, I had to use an external library to make the call. I used the needle library as the Axios library failed for other reasons. Needle correctly registers the webhook and returns the webhook id for use in a subsequent unregister call. The problem I have now is that my application, that works fine in development, will not install. The system tells me that there is an error but gives not further indication as to what that might be. It simply tells me to call support, who have told me that they are not interested in helping to identify th problem and will provide no further information. This dos not bode well for any future development efforts.
Supposing I can figure out why the registration does not work and fix that, at this point, I do not know exactly what the actual webhook payloads from Pager Duty will look like. I cannot run this in test mode because Pager Duty will not call the URL that I am providing in the test system and I cannot get the installed code to run on the server.
A simple application that should have taken a couple of days to put together has taken three weeks so far with apparently no prospect of being completed satisfactorily. As you can imagine, my customer is not exactly pleased with this, or the response of the Freshworks support organisation.
I don’t care about the API Key to either system as they are both trial installations with no customer data in either one.
The error is being returned when the app is installed. I assume it is coming from the OnAppInstall but I do not see how to find any logs being produced. I contacted support and they said that no logs were available for this, which I did find a little odd. Can you point me to where I might find a log to perhaps see what is being rejected? The code works fine when run locally.
Since the platform does not allow other content types I am using the needle library to make the call. When using a GET to fetch incidents from PagerDuty the content type returned is application/json and hence this works, as pointed out by others. The problem is specifically in the area of webhook registrations where PagerDuty returns a different content type even though in reality it is perfectly compatible with application/json.
TL;DR Pagerduty APIs will not work with Freshworks apps if they send vendor-specific custom content-type response headers. If you already have a proxy/middleware to intercept these and act as the layer in between Freshworks Dev Platform and Pagerduty, you might be able to skip writing an app and directly talk to the Freshservice REST API. We have taken notes of the experience you have been through. Please accept our apologies. We will begin discussions internally on whether we can relax the restrictions around content-types in the near future.
Thanks for the detailed notes, @IanAber. This gives me enough clarity to work with. My apologies for the delay, the poor experience you have faced, and having to go in circles to access the right help. We are aware of these loopholes and are working towards building a better support experience for our developer community. I appreciate your patience.
This makes sense. I had tried only the GET requests. As you know already, request method has these known limitations around content-types. So, it will reject responses that do not have one of the accepted content-type headers.
These design decisions are usually based on security. We will investigate the design decisions further to see if we should consider removing this restriction or allowing a developer to declare trusted content-types.
This is, unfortunately, a gap in the user experience today. We are working on making error messages clearer and hope to have a glossary to look up all the errors soon.
Yikes. This is exactly the kind of experience we don’t want our developers to face. Once again, sorry that you had to go through this.
If you have console.log() statements in the onAppInstall event handler, you should be able to see those logs by logging into the account where the app is installed, go to Admin → Apps → Manage Apps, choose the app you want the logs for, then click Settings → View logs.
It is very likely the incoming webhook payload will be rejected if it does not have one of the supported content-types.
This is another design decision that we need to review on priority because while you can still use a 3rd party HTTP client in a serverless function, there is no way to bypass this restriction for incoming webhooks (external events).
FYI @Developer-Platform.
Thanks for your detailed response.
I have two remaining questions:
-
In the developer forum I found a couple of articles where developers had been told to use Axios to make external calls. It states that this will not be allowed in market place apps without special consideration from development but it suggests that for individual customers own implementations it is acceptable. Axios is a third party http library and although, in this case, it does not work, the Needle library does, at least in test. Since you say that the content type is most likely preventing the operation, does this mean the this applies to any custom library too when in production? What is the full scope for using any custom libraries here?
-
Given that this error occurs on installation of the serverless application thus preventing it from being installed, how can I select the application to examine the log? There is no application to select.
vnd.pagerduty+json is officially registered with IANA.org as a valid content type as are thousands of others that would be excluded but Freshworks. Just a quick scan suggests that there are many systems that are being excluded from integration possibilities with Freshworks because of its intolerance to these official types. Instead of an arbitrary hard coded list of acceptable content type I would respectfully suggest that Freshworks considers the following approach.
- Allow all officially registered content types to proceed. Registered content types can be viewed here https://www.iana.org/assignments/media-types/media-types.xhtml. I suspect that there is an API where this information can be queried automatically and kept up to date as new types are registered.
2.Allow the application to specify the expected content type and pass that on if it matches.
By restricting the content type as much as is currently being done Freshworks is excluding a great many possibilities for integration with external systems using serverless applications and potential forcing customers to use a server based approach with middleware to provide the interface thus negating the benefits and distinct advantages possible with the serverless architecture.