ITOM Jobs stuck in Processing Output

[AndyGilly]

Afternoon 

we are getting intermittent site of jobs getting stuck in the Processing Output status. Its not just the job below, it seems to be across all

If you re-run the job type they complete successfully

It would be great if this could be looked into and we could understand if there is any monitoring that could let the supporting team know if jobs get stuck??

 

thanks

Andy

[Ricky]

Hi@AndyGilly,  I believe that this is an issue we are currently aware of and is in the development queue for further investigation.   I will enquire into any progress  they we have made and request that it is given a higher priority.  Can you confirm that the issue is consistently exhibits that same behaviour?  In that initially it stalls and subsequent executions are successful.  Also does refreshing the GUI in the Job queue have any effect?

Regards,

Ricky

 

[Ricky]

@AndyGilly, Can you review the SIS log (EspSisService.log), and check that the following lines exist for any of the stalled jobs:

2021-03-10 16:52:00Z [INFO]:[GENERAL]:[1164] Result: Success
2021-03-10 16:52:00Z [INFO]:[GENERAL]:[1164] Setting job 317 status to processingOutput
2021-03-10 16:52:00Z [INFO]:[GENERAL]:[1164] Setting job 317 status to succeeded

Regards,

Ricky

[AndyGilly]

Morning @Ricky

below is what i can see in the log for the completion of that job

2021-03-10 16:53:09Z [INFO]:[GENERAL]:[5340] Result: Success

2021-03-10 16:53:09Z [INFO]:[GENERAL]:[5340] Setting job 317 status to processingOutput
2021-03-10 16:53:09Z [INFO]:[GENERAL]:[5340] Setting job 317 status to succeeded
2021-03-10 16:53:09Z [ERROR]:[COMMS]:[5340] The SIS authorization key is invalid
2021-03-10 16:53:09Z [ERROR]:[GENERAL]:[5340] Failed to update job status. Operation[automation::sisUpdateJobStatus] The SIS authorization key is invalid

thanks

Andy

[Ricky]

Hi @AndyGilly, looks like the SIS authorization key is being reported as invalid for some reason, our dev team are looking into the issue and I will get back to you as soon as I get a response.  

Do you have more than one SIS and what is the pairing status of them in the Admin tool?

Regards,

Ricky

[AndyGilly]

Hi @Ricky

only one SIS our end

thanks

Andy

[Ricky]

@AndyGillyThanks for the update. Have spoken with the Dev team and they suggest that the issue must be the result of some form of communication error.   I am currently in the process of organising access to the logs from yesterday, in order to investigate this further.   I may also require access to the logs located on the SIS sever on you're network, and may require them to sent over.  Due to the nature of the investigation I will be requesting that an Incident be raised on your behalf to track future progress, and will advise you of the incident reference.

Could you also confirm roughly when you first started to experience the reported behaviour?

Regards,

Ricky

[AndyGilly]

Morning @Ricky

thanks for the update

We have only really started to put a higher volume of jobs through ITOM in the last couple of weeks and therefore the issue has been noticed more.

Although, I do think I have seen this behaviour pretty much since we started using ITOM

thanks

Andy

[AndyGilly]

Hi @Ricky

is there an update on PM00166852 please???

we have been doing some new starter testing today and have found the issue to be appearing more frequently. We cannot see any logic to the error occurring (differing jobs etc)

thanks

Andy

[Ricky]

Hi @AndyGilly, I will chat with the dev team and get an update on any progress.

Ricky

 

[AndyGilly]

Hi @Ricky, polite request for the latest on this one,

thanks Andy

[Ricky]

Hi @AndyGilly, Apologies for the delay in responding. I am still waiting for an update on this issue. I know that we have been able to identify the root cause of the problem and that the fix will be within the SIS. A release is due soon; however, I am still waiting for confirmation of the timeframe as I believe dependencies are holding back the release. I hope to get more information and hopefully an ETA later this week, and I will update you accordingly.
Regards,
Ricky

 

 

[Ricky]

@AndyGillyI am still waiting for confirmation of the next version of the SIS release date that will contain the fix for the issue you are experiencing. I believe that there were discussions on this late last week, and I am still waiting for further details. Apologies for the delay, and as soon as I get some information, I will let you know.

Regards,
Ricky

 

[AndyGilly]

Morning @Ricky

it would be great to get an update, we are holding off putting our automated new starter process live until this fix is in & and I would like to plan our release if possible please

thanks

Andy

[Ricky]

Hi @AndyGilly, I am still waiting for confirmation of a date when the fix will become available, but will give the dev team a poke to get some progress with this.

Regards,

Ricky

[Ricky]

@AndyGilly, Have spoken with the dev team and they are hoping to get an SIS release out next week, which will contain the fixes required to resolve the issue.  I will post an update as soon as I can to confirm that the update has been pushed for release.

Regards,

Ricky

 

[AndyGilly]

Hi @Ricky how we looking with this one??

thanks

Andy

[Ricky]

Hi @AndyGilly,  It looks there were issue identified during the testing of the fix, and Dev. are currently looking at the issue.  I will give then a nudge and get back to you once they have responded.

Regards,

Ricky

 

[AndyGilly]

Hi @Ricky

appreciate an update

thanks

Andy

[Ricky]

Hi @AndyGilly, apologies for the delay on this. I have spoken to the Dev, who has confirmed that the fixes are in the next SIS release, as mentioned previously.   We hope to see the release at some point next week, and once I have a more precise release date, I will update accordingly.

Regards,

Ricky

 

[AndyGilly]

Hi @Ricky just checking in on the next ITOM release progress??

thanks

Andy

[Ricky]

Hi @AndyGillyI am hoping for some movement on the release this week, it has been pushed to our test servers and assuming there are no issues should go live.  I will give the Dev team a nudge to see if I can get a more accurate time frame.

Regrads,

Ricky

 

[Ricky]

Hi @AndyGilly, I can confirm that we have pushed the core changes to build yesterday and should be available on Friday, given that there are no significant issues identified during the final test phase.

Regards,

Ricky