enhancement Webhook request sub-entity payloads inconsistent

[Gareth Cantrell]

I have noticed that the update payload for Incidents (urn:webhook:entity:Incidents:updated) are not consistent with the other Request entities (ChangeRequests, Problems, Releases).

For example, I have a webhook listening for updates to Change Requests and the webhook sends the h_fk_reference field in the record structure:
 

{
  "onEntityEvent": {
    "eventSource": "urn:webhook:entity:ChangeRequests:updated",
    "callingSessionId": "U2023051500000957.5086",
    "eventTime": "2023-05-16 11:05:35Z",
    "actionBy": "xxx",
    "actionByName": "xxx",
    "entity": "ChangeRequests",
    "record": {
      "h_fk_reference": "CH00000234",
      "h_custom_a": "Medium",
      "h_last_updated": "2023-05-16 11:05:34"
    }
  }
}

However, when using Incidents, the webhook sends the h_pk_id only, so it is not possible to determine the request Id as the API doesn't appear to have any methods for retrieving the underlying request:

{
  "onEntityEvent": {
    "eventSource": "urn:webhook:entity:Incidents:updated",
    "callingSessionId": "U2023051500000956.4189",
    "eventTime": "2023-05-16 10:28:40Z",
    "actionBy": "xxx",
    "actionByName": "xxx",
    "entity": "Incidents",
    "record": {
      "h_pk_id": "121",
      "h_custom_a": "Medium",
      "h_last_updated": "2023-05-16 10:28:40"
    }
  }
}

I can get around this by configuring the webhook to send a Requests:updated event, however this is inefficient and results in more outgoing calls than is necessary when I'm only interested in Incident updates.

Is it therefore possible to update the Incidents payloads to include the h_fk_reference field, rather than the h_pk_id field?

[AlexTumber]

@Gareth Cantrell thanks for your post.

Unfortunately it's not possible for us to configure the payload on an entity by entity basis. At this current time, the web hook functionality sends the primary key plus any columns that have ben modified for all application entities.

Personally, I would love to see us update the incident table and have a consistent set of columns across the board but a change like that to a major table that's used by every one of our customers involves a lot of work (to the detriment of other features/functionality), as I'm sure you can appreciate. The fact there is also a viable workaround available (admittedly not ideal) is also likely to mean this won't be addressed anytime soon unfortunately. 

I will post back to this thread if there are any updates or any movements on this particular known issue.

Alex

[Gareth Cantrell]

@AlexTumber thanks for the update.

I can appreciate that it'd be a major change and we can live with the workaround.

On that note, it would be really nice to be able to attach conditions to a webhook, insofar as it would then only fire if those conditions are met.

Currently, a single update can cause a webhook to initiate multiple calls, with varying information carried in each call.

[AlexTumber]

@Gareth Cantrell no problem. I totally agree and adding conditions to web hook events makes a lot of sense to me. 

I'll forward this requirement onto the relevant team at Hornbill.

Alex

[Gerry]

@Gareth Cantrell

In the roadmap and under active development is something we are calling business rules.  Different to web hooks, but will have conditions that trigger actions, so some of what you are doing with webhooks you will probably be able to do with business rules, but should be easier to maintain. 

To be fair though, conditions on web hooks is actually not a bad idea for consideration. I am always conscious of over-complicating Hornbill when we consider features, but WebHooks are really an advanced feature of our enterprise offering so I don't see why we could not add conditions to web hooks.  From memory, everything we would need to be able to apply a condition check over the record relating to the web hook is in our hands and we have an ultra fast expression evaluator that could be applied to that exact data.  I will add a review of the code to see how viable that is. 

In relation to the question about the events, in your example, I believe the event for ChangeRequests is actually being fired as a result of a Requests entity update, meaning the API that was called that caused the ChanegRequests to fire a webhook was in fact an API call to update the Requests entity, which has the ability to update a related entity indirectly.  So the record being updated is a related record, hence the FK value, as opposed to the requests primary key.  It *may* be possible to provide the primary key value of the main entity that was updated under these conditions, I am not exactly sure, again I would need to look and see whats possible without breaking things or sacrificing efficiency.  Its not likely the code change that will be the issue, its setting up the replication and test environment, and the subsequent add of automated testing, so time is needed here.  

I will post an update once we have had a look at whats possible here.

Gerry

[Gerry]

@Gareth Cantrell

Following a code review, I have added an item to our 90 day commit backlog for WebHook expressions. In essence, for each webhook there will be an option to specify an expression. If no expression is specified (the default) then the webhook will always trigger.  If you specify an expression, the expression will be evaluated before triggering the webhook, and, if the expression evaluates to false, then the webhook will not be fired.  Its most likely that the expression will be limited the fields that you see in the webhook event message.  The expression is SQL-like syntax, so the same as what you would typically see on the right side of an SQL WHERE clause. 

I am not sure how useful this will be in real life, but so much of whats required to implement this is already in place, so its a very simple code change and does not need to sacrifice any performance, and creates the opportunity to run more efficiently if you only ever need to trigger certain web hooks dynamically based on certain data values,  so is a no-brainer of an addition. 

Watch out for the change in the platform release notes in future weeks. 

In relation to the other question about getting extra information when firing a related entity webhook.  As I expected, this is much less self-evident, the code paths are complex and so we will have to create an end to end set of tests in order to carry out an evaluation of what could be done to improve things here.  However, again, a simple amendment to the existing behaviour is to add an option to include "non-modified" record data in the webhook record payload, so this may well resolve your immediate question as it should include both the record primary key, and the FC values. 

There is one further option included in this change.  This needs to be enabled, but basically this is an improvement to the JSON payload. Historically we have supported XML, but at some point we added the ability to choose XML or JSON as the webhook payload data format, but did not address the fact that while XML serialization relies on XML Scheme to determine datatype representation, JSON has limited datatype representation, in layman's terms, this means, from a JSON payload you can differentiate between text, integer and numeric values. We cannot enable this by default because that would be a breaking change, but there will now be an option so you can use the native JSON serialisation if you want to. 

Gerry

[Gareth Cantrell]

@Gerry

Thanks for the feedback! As Bob will attest to, I have been fixated on the business rules feature ever since I saw it in the roadmap, and it will hopefully allow me to retire my brand new Azure function apps before they get too old

I am happy to hear about the addition of expressions to Webhooks. Cutting down on the extra logic I currently need in my scripts will make it easier for my colleagues to manage these (until the business rules feature drops). 

[Gerry]

@Gareth Cantrell

By way of an update, webhooks now include a couple of changes that should help out with what you are doing.  

1. You can now add an expression to dynamically control when a webhook triggers, based on the data that triggered the webhook.   The UI includes a syntax highlighting editor and some basic intellisense support for when you are creating the expressions. 
2. There is a new option, that if enabled, will send all values in the record in the webhook payload, and not just the saved fields, this should get you the primary for a record when its being updated indirectly as a related record from its primary record. 
3. There is a new option that, if enabled, will use correctly formed JSON in relation to non-string values.  Because of a hang0over of using XML without any schema, values in the payload are all strings, this option will switch that to use JSON native integer, numerics and booleans instead of putting those values in quotes. This has to be an option because of backwards compatibility.

This should all be available in the next 2-3 weeks. 

Gerry