Completed BPMs Retention Time

[will.good]

HI,

In the interest of storage space, how long are completed business process kept for, and is there a way to set a retention period for them? after which they will be deleted?

[Berto2002]

The Workflow instance that is created for a Request is removed, I think along with its log, 7 days after the Request reaches the end, assuming it's not in error. There are quite a few posts around from HB staff confirming this.

The way to ensure they don't stay around longer is to have a housekeeping regime to check all workflows reach their full and proper end node. Any left in a perpetual state of suspended, hold, or in failed state will stay around forever and use-up storage.

For example, one of our reports looks for those in closed statue with BPM status of 4 (suspended). If a request is closed and the BPM is not ended then we have a workflow issue so we analyse the requests and then fix the issue allowing a close before it ends and then cancel/manually address the offending tickets.

[will.good]

Interesting. We do regularly check any that error and resolve these.

We have nearly four thousand pages of completed workflows dating back to 2022, so I'm not overly confident that they remove after 7 days...

[Berto2002]

I stand corrected. I thought the BPM/workflow also was removed but it seems it's not.

I have also gone way back and can see each BPM/workflow attached to old tickets.

There appears to be no process for the BPM to be removed.

[Berto2002]

Old documentation on storage: Managing Instance Storage Usage - Hornbill which I hope will be migrated over at some point to new KB.

I think there is an opportunity for an enhancement here @James Ainsworth. At present only the logs seem to be deleted after 7 days but we could set a retention also for the workflow (aka BPM) to be deleted from Completed Requests. Hornbill and customers could save a huge amount of storage (costs) since the BPM config files can be Mb each?

[James Ainsworth]

7 hours ago, Berto2002 said:

Old documentation on storage: Managing Instance Storage Usage - Hornbill which I hope will be migrated over at some point to new KB.

Yes, I have this flagged for moving to docs.hornbill.com.  Hopefully we will see all the remaining pages on the wiki moved over soon.

[Berto2002]

And how about the idea that much data could be saved if the BPM itself was removed from the Request after a certain time, like the Log is?

[James Ainsworth]

1 minute ago, Berto2002 said:

And how about the idea that much data could be saved if the BPM itself was removed from the Request after a certain time, like the Log is?

I know that the list of main Workflows is counted toward data usage.  I have asked the question if workflow instances are included.  The documentation here: https://docs.hornbill.com/hornbill-cloud/data-storage simply says "Business Processes" so I wanted to clarify what that includes. I also don't know what the implications of deleting a workflow instance.  This will remove the workflow tracker at the top of the request so once deleted, if you were to go back to an old request, you wouldn't have visibility of the stages or checkpoints that the request went through and possibly no indication of what workflow a request had used. 

[James Ainsworth]

Just to confirm that the workflow instances do count toward the data quota.  The question from here possibly turns to value and if there is value in maintaining the workflow tracker at the top of the request for historical reasons vs saving some space.

[Berto2002]

@will.good, @Adrian Simpkins, @HGrigsby, @Gareth Cantrell @EWA would be interested in your thoughts about this.

@James Ainsworth When I download my two most common workflows (generic incident and generic request), they are 1.5Mb approx in txt file. Visually, see screenshot for how one of these standard workflows looks for reference of what 1.5Mb is (plus one smaller stage for closure). We get through 200 of these a week so are we really talking about 300Mb? Other customers much more of course. I guess database compression etc makes storage much lower but how much lower?

James, can you please ask Devs is that is the size of a retained BPM against each Request vs it's text file size (or other stats like the percentage of storage taken by a workflow vs the request data itself)? If this is a significant number (and I think it must be if the full integrity of the workflow is maintained for every historical Request), there will be a clear case to enable discarding the workflow/BPM according to a system setting and would likely have it set for 1 year +. Historical tickets for us are really only ever about the Request data and are never looked at in order to see HUD information.

I would venture that the space saving opportunity here is so significant for HB customers and HB that it may even be worth coding a 'snapshot' view of the HUD which is set as part of the BPM/workflow removal process; so the HUD remains but as a static view.

I remember at HUG someone mentioned you were considering more about the data retention now the system is maturing; and this could be a great enhancement to put on the list.

[will.good]

9 hours ago, James Ainsworth said:

Just to confirm that the workflow instances do count toward the data quota.  The question from here possibly turns to value and if there is value in maintaining the workflow tracker at the top of the request for historical reasons vs saving some space.

Thanks @James Ainsworth,

We'd like a way to be able to either set a retention period or a select all to be able to delete them. We don't use the HUD on requests so would not be missing this and if we need to refer back to a request we just use the timeline.

It seems like we could have quite a significant space-saving by not keeping these which we would value more.

[Gareth Cantrell]

@Berto2002 from my perusals through the system, an instance of the workflow is not created for each request, rather, a link to the specific workflow version is stored in the bpm_instance table.

If you make modifications to an in-process workflow instance, only then is a copy of the original version taken from the bpm_processes table and the modified workflow stored in the bpm_instance table.

So, the only storage is each version of your workflow in bpm_processes and any modifications made to a live instance.

For data retention purposes, I don't believe I've seen any option to delete historic versions of a workflow.

[Berto2002]

@Gareth Cantrell that is very useful to know and it makes sense as a design. So the only storage issue we may therefore have is the 245 versions of our generic workflow

[will.good]

4 hours ago, will.good said:

Thanks @James Ainsworth,

We'd like a way to be able to either set a retention period or a select all to be able to delete them. We don't use the HUD on requests so would not be missing this and if we need to refer back to a request we just use the timeline.

It seems like we could have quite a significant space-saving by not keeping these which we would value more.

Could one of HB Staff tag this as an enhancement based on this comment please?

[Gareth Cantrell]

1 hour ago, Berto2002 said:

@Gareth Cantrell that is very useful to know and it makes sense as a design. So the only storage issue we may therefore have is the 245 versions of our generic workflow

I have a handy little SQL query that can give you a rough idea of storage requirements for workflows (incl. autotasks) if you have access to the Database Direct tool:

SELECT count(*) AS "amount", 
        "modified instances" as "type",
        sum(length(h_xml))/1048576 AS "size_mb"
FROM h_bpm_instance
WHERE h_xml is not null
UNION
SELECT count(*),
        "workflows (inc. all versions)",
        sum(length(h_xml))/1048576
FROM h_bpm_processes

 

[will.good]

On 25/04/2024 at 13:26, will.good said:

Could one of HB Staff tag this as an enhancement based on this comment please?

@James Ainsworth

 

[Berto2002]

On 25/04/2024 at 13:47, Gareth Cantrell said:

SELECT count(*) AS "amount", 
        "modified instances" as "type",
        sum(length(h_xml))/1048576 AS "size_mb"
FROM h_bpm_instance
WHERE h_xml is not null
UNION
SELECT count(*),
        "workflows (inc. all versions)",
        sum(length(h_xml))/1048576
FROM h_bpm_processes

"MySQL Error 1969: Query execution was interrupted (max_statement_time exceeded) SELECT count(*) AS "amount", "modified instances" as "type", sum(length(h_xml))/1048576 AS "size_mb" FROM h_bpm_instance WHERE h_xml is not null UNION SELECT count(*), "workflows (inc. all versions)", sum(length(h_xml))/1048576 FROM h_bpm_processes limit 1000" so I needed to set the rows to 25000 but then got this:

So @Gareth Cantrell Can you describe in words the 'meaning' of the modified instances and workflows here please? Our storage utilisation is about 40GB in total so it looks like up to a quarter of that could be superfluous; and that would be a massive saving for Hornbill and Customers if it was managed.

[Gareth Cantrell]

Hi @Berto2002,  "Modified Instance" is an instance of the workflow that has been modified AFTER a request has been generated.

Basically, if you go into a Failed workflow instance (or any running workflow instance, for that matter) and make a change, the changed workflow is stored in the h_bpm_instance table in the h_xml column.

If you don't make changes, the h_xml column in the h_bpm_instance table remains empty.