[SEVERE] Hornbill instance performance and connectivity issue (RESOLVED)

[Lyonel]

We seem to be back to normal here too 

[Guest]

The web app is working fine but we're still seeing an issue where emails look like they are being picked up by Service Manager but not put into its inbox.

[Victor]

53 minutes ago, james.edwards said:

we're still seeing an issue where emails look like they are being picked up by Service Manager but not put into its inbox.

@james.edwards I had a quick look and to be honest I don't see any errors or issues that could cause this... can you give me a bit more details on this? How do you know the emails are picked up by Service Manager?

[Guest]

@Victor Thanks for taking a look. I'm presuming they are based on no obvious errors in the log files but could easily be wrong. I'm also speaking to hosting partners just to check if it's with them or not. Running what testing we can here. We can receive emails sent from Hornbill to internal addresses and the mailbox (and Hornbill) is receiving emails sent via external routes. E.g. Contact forms on our website that get passed through. We can’t seem to send mails to the address internally.

[Victor]

7 minutes ago, james.edwards said:

We can’t seem to send mails to the address internally.

@james.edwards are these emails sent from an email address within your domain to Hornbill mailbox?

[Ralf Peters]

Hi @Victor ,

we are noticing a change in our BPMs since yesterday's issues , in particular  , the automated task  "waiting for request resolution" , the action focus does not  go to the resolution window anymore  if left on auto ? Just wanted to check if that is a new "feature"   before i go and change them to manual  and resolve .

Thanks

Ralf

 

 

[Victor]

@Ralf Peters the default focus has always been on "Update" unless specified otherwise by a node... so to have focus on resolution during "wait for request resolution" the node would need to be manually set to focus on "Resolve"

[Ralf Peters]

Hi @Victor , strange , just   check a number of our BPMs all have the action focus set to Auto   in Automated Task - Entity-Requests-Suspend-wait for request resolution   , all were focusing on the resolution window until yesterday afternoon.

if i hover over the info icon  of "Action Focus" is says that the default if unspecified is the resolve action ......

 

 

[Victor]

@Ralf Peters I am quite sure our dev team advised me of the behavior I described... but I can see the node description suggests something different so I'll raise this internally...

[nasimg]

Hi @Victor

Can you check our instance, I have a number of analysts reporting slow down and I've seen the "Failed to initialize application Operation timed out after  10007 milliseconds with 0 out of -1 bytes received" error when trying to open tickets (eg SR00135213)

Nasim

[Dan Munns]

@Victor is was under the impression (and our instance certainly seems to confirm this) that a 'Suspend - Wait Resolution' set to auto focus will focus on the resolution window rather than the update window.

Indeed all suspend actions seem to auto focus on the relevant suspend window (apart from logging category which I have raised before) 

[Kelvin]

Are we still experiencing issues as we are still getting a lot of disconnects error screen, typically it is back up at the moment so unable to get a screen shot

[AndyColeman]

We have also been experiencing issues since around 20 minutes ago. 

[Kelvin]

Just happened again

 

[Guest]

18 hours ago, Victor said:

@james.edwards are these emails sent from an email address within your domain to Hornbill mailbox?

@Victor Yes they're from within the same domain. The issue has been identified and sorted with a hosting partner not renewing certificates in time, so for you and everyone else NOT a Hornbill issue. Our instance seems to be running fine now.

[Victor]

@Kelvin @nasimg @AndyColeman we're investigating the timeout issue. Everything seems fine now. Do you still experience the timeout issue?

[Victor]

@Ralf Peters @Dan Munns the incorrect action focus has now been confirmed as a defect, our dev team is working to have it fixed.

[nasimg]

24 minutes ago, Victor said:

@Kelvin @nasimg @AndyColeman we're investigating the timeout issue. Everything seems fine now. Do you still experience the timeout issue?

Seems better for us now - but what happened. I can't keep telling staff everything is ok only for it to slow down/error. I need to be confident you have located a fault and resolved it. If not, tell me you didn't find anything and everything is behaving normally now.

When I reported it (and the others) it was consistently bad, so was any thing visible on your side?

[Victor]

@nasimg we don't know what happened yet I'm afraid... as I mentioned before we are investigating this. The reason I mentioned all should be fine now is that our monitoring tools are reporting all functioning within parameters. We will locate the fault and fix it. I do apologise but there is no other information I can relay at this moment other than this

[Victor]

@nasimg @Kelvin @AndyColeman our investigation so far revealed an unusual spike in DB resource usage at the time the issue occurred today. This is why you experienced the timeouts and disconnects. As an immediate measure, we will increase the available resources tonight. We are still looking to find more information on this.

 

NOTE: today's issue is not directly related to the issue reported in this thread yesterday.

[AndyColeman]

Thanks @Victor

[m.vandun]

@Victor
We're experiencing problems as well, where activities are not being created and the auto focus is not working.

Kind Regards,

[Victor]

@m.vandun

1 hour ago, Victor said:

the incorrect action focus has now been confirmed as a defect, our dev team is working to have it fixed.

Regarding the tasks not being created, may I ask to raise it separately? There were (and are) different issues reported and discussed in this thread which dilute the original issue for which this thread was raised.

[nasimg]

Hi Victor

I'm still seeing performance issues on our instance, and getting complaints from the analysts that things are really slow.

Can you check what is currently happening as it should be relatively quiet at 17:20.

Nasim

[Victor]

@nasimg this:

 

We are working to have it fixed now