[Enhancement] Hornbill Automation for Simple Lists

[samwoo]

Good afternoon,

I would like to request for a couple of additions to the Hornbill Automation, the ability to add or delete Simple List items.

I can do this easily using the HTTP Request node, but feel it would be better if it becomes a Hornbill supported addition to the growing list of Hornbill Automations.

Add List Item

Body

Headers

URL
https://mdh-p01-api.hornbill.com/<INSTANCE NAME>/xmlmc/data/?op=listAddItem

[Gerry]

Hi @samwoo

At first glance we are not sure this is very wise, I will expand on that, but before I do, can you please give us an outline of the use case here, and why you need to dynamically change simple list content via the BPM?

Thanks
Gerry

[samwoo]

Hi @Gerry,

Our Data Team are looking to utilise Release Management and Change Management for Date Warehouse workflows.

The idea is to allow customers to raise individual SRs for scripts to be used to load the data into the Warehouse and for members of the Data Team to log one or more release tickets and go through the tickets logged by the customers and using the BPM of these workflows, once they pick up the ticket as an "Owner" a Human Task appears allowing them to associate the SR with any of the available release tickets. There are a small number of people in the Data Team and they may each raise a release ticket for the current week, or for future weeks so they can plan their workload and prioritize accordingly.

Once the release is selected in the SR's Human Task, the release ticket is then automatically linked to the SR (I haven't gotten this bit working yet). Other fields on the SR such as the External Reference (even though its internal) is populated with the Release Ticket reference.

When a Release ticket is logged via a certain Service and Catalog with bespoke IC and BPM, I currently use the experimental HTTP request node and the Hornbill API to insert the release reference number into a simple list. Once the bpm reaches a certain part of the release workflow, the release reference number is removed from the Simple List using the experimental HTTP request node, meaning that it's no longer possible to associate any further SRs with the release (an existing release would need to be selected, or a new release would need to be logged for a different week), and the current release can then move on to the next stage of implementation and UAT.

Our general users don't have access to the configuration side of Hornbill, and we would like to keep it that way (for obvious reasons), and between myself and my colleague who supports Hornbill, we also support other applications as well, so we cannot maintain Simple Lists each time there is a release (and the Data Team are a different part of our Service).

This is the reason for requesting automation in this area.

If the concern is that it can impact other simple lists, then could we have security measures applied to Simple Lists that determine which ones can be updated via BPM and which ones cannot? Also prevent Hornbill standard ones from being manipulated via the BPM as well?

The other idea I just thought about is what about the possibility of having a Dynamic Dropdown List Box Data Query that can be used return a list of Request References (paginated) but we can add filtering to it to narrow down the list. The only problem with this is that Data Queries are not possible via Human Tasks.

* I had almost forgotten about this, but I have also got a Power Automate flow that update simple lists via a Webhook when a mobile phone asset is created, and the sub state is "available for customers".
This list is used in a Human Task on one of our processes where the officer can select one of the available asset numbers, process the ticket and mark the asset as assigned to a user, the bpm will then remove it from the list and I've not actually had any feedback about it (hence the reason I almost forgot about it) but it appears to be working fine. Again, this is done because Data Queries cannot be used in Human Tasks.

I hope this makes sense; it's been quite a long morning for me so feeling drained but wanted to get this response sorted for you.

Thanks,

Samuel

[MacLean Ferguson]

We've also been looking into using BPMs to add or remove items from simple lists. I'm interested in why that wouldn't be wise for us to continue considering as a solution. 

[Gerry]

@MacLean Ferguson

Quote

I'm interested in why that wouldn't be wise for us to continue considering as a solution

Thanks for your question.  First, I do want to say that we are delighted that customers find new and imaginative ways of getting more use, and more value out of Hornbill, we are all for that.  Its just when we see things that might lead to things breaking in the future. or might lead to unpredictable results then its worth thinking about.  Sam has asked for some features that will help facilitating the dynamic changing of content in simple lists via a workflow, and in the absence of a ready-made Hornbill Automation to do that, he has found a way of making an API call via the iBridge back into the instance to achieve that, but is asking for a better way.  

Sop now we know he has a need to do that, what we do not know is whats the use case, because perhaps there are better ways to achieve what he wants.

Ok so there are a number of things from the above use case that Sam has deployed that stopped to make us think...

• He has used an HTTP Request method in iBridge that is clearly marked as Experimental, that means that capability could actually be removed, or changed without notice at any time - not a very good thing to depend on in a production environment. 
• Simple lists, are by design, an "customisable via administration" static list of data.  These are designed that way, and while a simple list on the face of it is just a bunch of small records in a database, the reality is quite a lot different to that.  Often, as in this case, we do highly optimise these sorts of lists, typical read many-write few caching semantics, which means, we load from the database and keep things like this in memory to keep database I/O and traffic low and efficient.  Now if a simple list is being used as a dynamically customisable list on a Workflow by Workflow instance, that changes the nature of that data completely, specifically the caching semantics are off, and we could easily end up changing our code in the future that could change this behaviour, for example, we might have delayed write semantics, in a distributed environment, this means there is no garuntee at all, that when you make an API call to change the data, that the next API call that hits a different process might not even see those changes. 
• Further more, simple lists are global, shared lists. They are not designed to be dynamically changed on a request by request basis, on busy systems you could easily end up with data loss or corruptions as far as any application logic is concerned because the write semantics will most likely be the last write wins. 

So the reason I ask the "what is the use case" quesitons is, I would rather none of these things happen to Sam, I would rather offer guidance that would enable him to avoid these types of situations.  Sam has worked out a way to do something he speicifcally needs to do, I am asking what does he specifically need to do, so we  can understand that, and consider if Service Manager should offer what ever he needs to do as a feature/capability so that he does not have to use his technical solution which feels like it could be a bit of a work-around. 

Now, if Sam says, we are using the automation to set up a new department and we need to add that department to a list of departments, or something like that, where you can see its more of an automated administration task, then fair enough, adding that to either a Hornbill automation (via Workflow or AutoTask, or even as an ITOM package would make perfect sense), but as I say, that does not appear to be what he is doing, which is why I have asked the question

Hopefully that all makes sense. 

Gerry

 

 

 

 

[MacLean Ferguson]

@Gerry

Thank you for all of that. You almost directly hit our use case at the end there. We are looking to use a business process to onboard locations and add them to simple lists based on the locations functions. We have a lot of pop-up locations come through various partnerships that we want to be selectable in our pharmacy catalogs if we deploy a pharmacist to the site. Being able to manage where each site shows up as we on and off board the locations would reduce a lot of the work in managing lists and reduce misses for us.   

[Gerry]

@MacLean Ferguson

In that case a simple Hornbill automation on the com.hornbill.core app is probably the right answer., but via the BPM there may well be a priv's issue, adding to a simple list requires certain rights, if the user does not have those rights and the BPM is running in the context of that user then using an API key to call back into the instance through the iBridge gives you more control, so in that case it would be better to add this to iBridge Hornbill automation. Probably sounds like we need both - I will ask the integrations team

Gerry

[samwoo]

Thanks @Gerry for taking the time to look into this. I wasn't sure if any of your responses after my last response relates to my initial query (as well as MacClean's) or if you were you were just responding to MacClean's use case (from what  I can tell, it's similar to mine but I tend to waffle a lot in my posts by) but hopefully what I have outlined is similar where the need is to just add or remove values on a simple list where required in the BPM, but in my case the simple lists resides in com.hornbill.servicemanager. iBridge sounds like a good solution.

 

 

[Gerry]

@samwoo

Ok so to summarise I think you are using the simple list to set a value a bit like an atomic global variable as a way of allowing one ticket to make its self visible to another ticket.  The problem I see with doing this is simple lists are really designed for standing data, so while you may be able to get something like that working, I would suggest its unlikely to be reliable for the reasons I mentioned above, so is not something I would be comfortable recommending, it feels like a hacky work-around for some kind of data token (request reference) sharing/gating.  

Let me ask you this, just in case I am not fully understanding your explanation.  If the perfect automation to enable you to achieve the outcome you are trying to get to existed, what would it be? what would it look like?

Gerry

[samwoo]

Thanks @Gerry.

The perfect automation would be to allow returning a list of references to a user in the form of a Data Query (available in both the Intelligent Capture and Human Tasks) with filtering capabilities.

For example, I just need a list of "new" and/or "open" ticket references against X service optionally raised via X catalog.

the raw value would be the reference number, the text value would be something like the reference number + summary.

Then the users (mostly via Service Manager in Human Tasks, but some may do it via the Employee Portal) can choose which of the available list of tickets (in my case would be Release tickets) they want to associate the work to and via the BPM the ticket will auto-link creating the parent/child relationship (there is another forum post regarding linking tickets via the BPM).

As we don't have main parent/child ticket fields I would store the release ticket reference number in h_external_reference and through the life of the child ticket (the SR), I can suspend and await for certain things to happen on the parent ticket (the release)

 

in my other example above regarding the mobile phone assets, again have a data Query that is both accessibl le in Human tasks and the intelligent capture to allow further filtering of the assets, so optionally filtering on X class, X asset type, X state, X substrate X operational state etc....

Selecting an Asset will return the asset ID as the raw value, and the asset name as the display value. Further to that we should then be able to carry out all sorts of asset automations on a single asset asset.

 

The biggest bugbear is the inability to use Data Queries in Human Tasks.

 

Hopefully I am making sense as I do waffle on, if it helps, I can send you the two bpms (relating to my first point) I have used to get this working with the current set tools that you have provided. I can suspend on the child SR tickets and wait for things to happen in the parent Release ticket quite nicely. Still a WIP though.

[Gerry]

@samwoo 

 

@MacLean Ferguson

 

Latest release notes to hit beta, simple lists can now be modified through Hornbill iBridge and Core automations in the Workflow if required. 

 

Gerry

 

[samwoo]

Hi @Gerry,

Many thanks for this, I saw this was available last week and all appears to be working as expected. 

Thank you!

[MacLean Ferguson]

@Gerry

This is great, thanks for getting this added!

[samwoo]

Hi @Gerry,

Just had a report from a user regarding the Simple List automation, they are getting this error message:

Never seen an error like this before. I was wondering if you or someone else could advise.

Thanks,

Samuel

[Steve Giller]

@samwoo We're aware of an issue in this area, this should be fixed early next week.

[Gerry]

@samwoo

Thanks, the error above is due to some recent security hardening we did, which inadvertantly impacted this area because of the way our private netowkrs work.  We will be resolving this, but, I would suggest, can you ask your user to use the Hornbill Automation for this instead of iBridge, that will be much better in the long run.  

Gerry

[Steve Giller]

19 hours ago, samwoo said:

Just had a report from a user regarding the Simple List automation, they are getting this error message:

@samwoo I had assumed from this wording that you were using the Hornbill Automation - if that is the case and you are still seeing this error then please let us know, but if you're using an iBridge (Cloud Automation) then as Gerry mentions you will need to switch.

[samwoo]

Hi @Gerry and @Steve Giller,

Thank you for getting in touch. I had been using iBridge as I cannot see any options related to Simple Lists under Hornbill Automation. 

Are you able to provide a screenshot to show where I need to look?

Many thanks,

Samuel

[CraigP]

I'm having troubling finding the Hornbill Automation version of this too.

I'm curious to know why the Hornbill Automation version is considered "much better in the long run"? Are iBridge automations not reliable?

[Gerry]

@CraigP

Quote

I'm curious to know why the Hornbill Automation version is considered "much better in the long run"? Are iBridge automations not reliable?

Thats a fair question.

Ok so the basic idea is this.  If you are automating your Hornbill instance, you should be using Hornbill Automations.  Why? because the security model in Hornbill is designed to ensure that you can do things, and more importantly can not inadvertently allow things to happen, in the case for example, where a user runs an AutoTask and has the right to run that, but one of the actions in the autotask is invoking something they do not have the privs to run.  When you run a Hornbill Automation its using the users security context. 

iBridge on the other hand is primarily intended for automating cloud systems that are external to your instance, these automations can include the automation of other Hornbill instances.   

In the above case, the iBridge is being used to automate (or if you like, call back into) your own instance. To make that work is more complicated for you because you have to create an API key, create a KeySafe entry, then set up the integration. 

Also, the design of iBridge was never really intended for self-integration, although it does work, we have no easy way of detecting/preventing rogue execution loops because the instance will see the iBridge API call as just another API call, there is nothing to make it aware that this is effectively an internal call.  In the future we may disable self-calling via iBridge for this reason. That is not actually on the cards now though because we have not run into a real-world production problem, but in theory at least that could happen one day

The tihng that spawned this thread was exactly this situation.  There is no intention/consideration given to the iBridge calling back into its own originating instance, and when we applied some security hardening we did not gave hany test cases for this scenario, the net result, we put the software update out, and a couple of customers things broke, because they happened to be using the self-calling via iBridge which was being blocked by the security change. 

So as a general rule, if we do not recommend something, or even recommend against it, then its generally because we think that future software changes may be impactful in the future. 

Hope that helps answer your question
Gerry

[Gerry]

@CraigP @samwoo

I am afraid I spoke to early, the Hornbill Automation has been added, but not rolled out yet, should be in the next core release, I can only appologise for the misdirection... you will see it in the next week or so. 

 

Gerry

[samwoo]

thanks @Gerry - I appreciate the .

I'll keeping a sharp eye for the release and will make the change promptly once it's available.