Issues with emails this morning? [RESOLVED]

[chriscorcoran]

Are there any issues with email being sent from hornbill this morning?

[Nanette]

@chriscorcoran  Yes, we are currently investigating with the highest of priorities, please bear with us.

 

[chriscorcoran]

Thanks! @Nanette

[Tina.Lapere]

+ 1 for us.  Well wait for the update.

[JonNutt]

Not seeing this issue on my instance so may be sporadic?

[Deen]

We are currently working on a hot fix, once this has been deployed to the affected instances I will post again shortly with an update.

[Adam Toms]

We are also experiencing issues this morning @Deen Any chance we could also be added into this? 

Many Thanks.

[Deen]

@Adam Toms We have raised an incident on your behalf.

[Victor]

@all

We have found the root cause for this issue and a fix is being rolled out on all affected instances.

[Deen]

The fix has now been rolled out to all instances.  Any queued emails should also be automatically resent.  We apologise for any inconvenience this may have caused. 

[Victor]

@all

It appears that we did not correctly evaluate the delivery state of the affected emails. These email were and are not automatically resent. We are looking to do a mass force retry which will re-send them.

Will come back with updates shortly.

[Victor]

@all

We have performed a forced resend on all failed emails in all instances affected by the issue this morning. If there are any more failed to be sent emails from earlier this morning (03/02) please let us know. 

[LifeOfJonny]

Hi,

I noticed this wasn't highlighted on https://status.hornbill.com/ which is where I normally go when something seems odd in hornbill, as the point in the site is to report any issues with hornbill but nothing was there.

[Victor]

@LifeOfJonny there won't be anything for the Status. This is reserved for availability issues where the service itself is not available or otherwise impacted. We define this as "a failure in the production operation of the service rendering the service unavailable or critically impaired". Although the email is an important functionality in Hornbill, and any issues with it are treated with the appropriate urgency, this was not an availability event. The service was operational and email functionality partially affected. And, I will mention again, just because an important issue is does not make its way to the Status page, for the above mentioned reasons, does not mean it is not treated with the appropriate urgency and priority.

[LifeOfJonny]

I shalt bother looking at that status site in the future shall just come straight to the forums then.

[Victor]

@LifeOfJonny I would not take that away from the above. If I may suggest, you could look at this if your service is available, if you can log in and use the majority of Hornbill functionality, then there won't be anything on the Status page and forum would be a place to go. If, however, your service is unavailable or you can't use Hornbill then that should be posted on Status page. Also you don't have to look at the status page (as in monitoring it). You can always subscribe to Hornbill Status page and you will be notified by email when an incident is posted there.

[RobW]

Are there still any issues with sending e-mails as I've just had reports that there's 50+ in our outbox here - all pending and appear to have had a number of attempts to resend. Seems intermittent as can send direct to myself from our servicedesk inbox in Hornbill if that makes sense.

Not going to lie, could have done without that this morning :-( 

[Victor]

@RobW Outbox or Sent Items?

[RobW]

@Victor - we have now have almost 60 messages sitting in the Outbox. Oldest appears to date back to 2/2. Have tried forcibly resending however attempt fails. I'm stumped - can raise with our infrastructure however don't recall any major 365 changes (none that we're aware of anyway).

 

 

EDITED BY ADMIN: Image removed - personal data (email address)

Edited February 4, 2021 by Victor
Personal data included in screenshot

[Victor]

@RobW there should be an error of sort in that log that mentions why it could not send the message...

[RobW]

Saw the screenshot had been deleted - sorry.

I appreciate there should be an error in the logs - didn't want to copy the whole thing for obvious reasons but I'm also trying to deal with some staff issues here while doing a compulsory training session. 

I really wish pubs were open.

[Victor]

26 minutes ago, RobW said:

I really wish pubs were open.

I don't think it resolves the problem but certainly one couldn't care less about them after the pints!