Outbound email failures

[Deen]

A number of users are reporting intermittent issues with outbound email stuck in the Outbox.  We are currently looking into the root cause and I will have a further update shortly.

[Oscar Stankard]

Hi there,

Thanks for the update on this and will stop testing things and wait for changes, it does jog my memory though please can we have the ability to set a FROM address as well as a TO address on the send test email function (my email server won't let me send as do-not-reply@hornbill.com)?

 

Thanks for your help and good luck in your troubleshooting of the problem.

 

Kind regards,

Oscar

[Deen]

After additional tests this appears to be an issue with Office365 where intermittently it is unable to complete a valid connection.  Any emails stuck in your Outbox will have 9 retry attempts, if any of these are successful they will leave the mailbox as expected - you will see from your Sent Items folder that emails are sending, it is a few specific ones that fail.  You can also manually resend affected mails using the envelope icon if they are time sensitive.

We have had reports from a number of other Customers with the same issue, so there does seem to be a problem with Office365 mail at the moment.

[Oscar Stankard]

Hi there,

 

I don't seem to have as much luck, have pressed the 'Resend Message to Recipient' option (the envelope icon?) but after doing that multiple times for every message I have still yet to see anything in the sent items folder past 9:55 this morning.

 

Do you have any observations about the success rate and how many times people need to retry for messages to be sent?

 

I don't seem to be able to find any information about a Microsoft change to their Exchange Online SMTP service or other people having issues so I'm a little in the dark about what's going on and what our best course of action is.  Have any further findings been made with the behaviour, announcements made by microsoft, or any further suggestions from Hornbill about ways of sending emails from Service Manager in the meantime can I check?

 

Thanks for your help.

 

Kind regards,

Oscar

[Keith Stevenson]

Oscar,
Thanks for the post. Sadly as this is a MS issue we are not able to comment on their reporting or issue. All we can say is that when we make a connection the TLS handshake fails.  

The below when you connect should allow you to say EHLO xxxxxx (Where XXXX is your servername in our case live.hornbill.com) . This hangs (and not just from our servers but from anywhere we have tried this) 

openssl s_client  -connect smtp.office365.com:587 -starttls smtp

If you compare this to say googles GMAIL you can see that this works as expected and on entering EHLO it shows list of options and allows you to carry on with the connection

 openssl s_client  -connect smtp.gmail.com:587 -starttls smtp

We are not a MS customer and therefore cannot raise the issue with them. 

As for alternatives, Direct Outbound rather than SMTP SMART host (See https://wiki.hornbill.com/index.php?title=Outbound_Mail_Routing) will prevent this as we go direct to the given mail server rather than via a 3rd Party who we have no control over

Kind Regards

Hornbill Cloud Team 

[Oscar Stankard]

Hi there,

 

OK thanks for your help, will propose switching to the direct delivery option in the meantime and see what the team says.

 

Is your direct to internet SMTP service able to submit messages to external domains on Office 365 at the moment?  Does the direct delivery option use the same connector as authenticated smart host delivery (and do Microsoft handle these TLS connections to the SMTP.office365.com and mail.protection.outlook.com with the same behaviour)?  Are there different results when using another cryptography library/tool?  How many other customers are still having this issue and are they geographically concentrated or with any other shared characteristics?  What have they managed to do in order to continue to send messages from their helpdesk?

 

While it's someone else's service that's at fault there is a great deal more information that would be helpful to both understand the state of play but also what our options are for resuming sending messages.  If this was affecting all ways of sending email via Microsoft's SMTP service I would have expected more reports of it or workarounds being drawn up, but struggling to find anything so anything that can be expanded on short of raising it with Microsoft would be very welcome.

 

Thanks for your help.

 

Kind regards,

Oscar

[Keith Stevenson]

Oscar
Thansk for the reply.  We have had no issues with direct email to Office365 hosted domains, the way that works is different. And the openSSL client is a open source tool not part of Hornbill (so outside). As for other customers I believe around 4% have reported issues, but given the intermittent nature of the issue others may not have seen a problem as they are eventually sent. 

1 thing about your logs is that on the occasions when MS does respond (its about 50% of requests ) they show a OAuth failure so you may have a secondary issue and you should check that (Although OAuth is not required for SMTP and MS are not removing it so we would recommend keeping Auth for Outbound as Classic ) 

Having just tried for 30 minutes attempting to connect and send different tools its about 1/4 success .. As Hornbill does the same thing every time we send an email (and in this case I was using the test connection option) this is a MS issue that is intermittent (most likely because they load balance the SMTP servers and only in 1/4 times do you get email sent.  If this was Hornbill we would expect 100% failure 


Kind Regards

[Martyn Houghton]

We too are affected by this issue as well.

 

[Blowerl]

We are affected as well with this issue.  

[jmcnae]

also affected - cannot find anything on MS issue on web - even had some emails sat in outbox where 1 recipient has been delivered and the other pending on the same email.

 

[Victor]

It is not very unusual for Microsoft to not have posted an update in this regard. We have raised the issue with Microsoft and we are looking into any possible alternatives and solutions that can be implemented on Hornbill side in the interim to restore the service functionality.

[Oscar Stankard]

Hi there,

 

We've got the direct delivery (DNS routing) thing going OK so people are happy enough in the meantime, don't think missing the outbound emails from our backups/archives is the end of the world in the meantime but there might be a way we can get an account created at Mimecast to allow authenticated 'smart host' delivery injecting messages into our normal delivery process but at a different point (and should then be captured on the way out and get backed up/archived again).

 

Would be good to understand if there is a different cryptographic utility/library that is able to connect to MS's SMTP since whatever it is changed yesterday morning and also what they say to Victor's request but pretty much ticking over here now glad to say.

 

Kind regards,

Oscar

 

 

 

[Victor]

@Oscar Stankard thank you for the update, happy to hear the Direct Outbound solution is suitable and working for you.

[Berto2002]

I confirm this has caused 20+ BPM failures this morning alone and we consider this a showstopper to our workflows. It's causing all external authorisation nodes to fail and we use this extensively for requests. I have logged with Support and appreciate this is something you are working on urgently with the supplier.

I understand that the External Auth sends from Hornbill core so I dont think we can implement a workaround from our end but can someone in HB please comment on this? I am engaging our tech to review also. 

[Martyn Houghton]

Switching over the Direct Outbound will require a DNS entry to be made  (if not already in place) and this then has a delay in propagation (up to 24-248hrs) to ensure the email is not then rejected by recipients email servers.

Therefore if there is anything Hornbill are able to do to alleviate the issue with the SMTP outbound issue with Office365 that would be a great help.

 

[Victor]

9 minutes ago, Martyn Houghton said:

Switching over the Direct Outbound will require a DNS entry

@Martyn Houghton yes, we know... it is a proposed solution, might not be suitable for some... we are definitely looking into finding more alternative solutions (if there are any) as well as any possible code changes we can implement here. I assure everyone this being investigated internally with the highest urgency and priority.

To clarify for anyone else, Direct Outbound requires an SPF record (which is in essence a DNS entry/record). More info here: https://wiki.hornbill.com/index.php?title=Outbound_Mail_Routing

 

[Berto2002]

We have added "Include:_spf.hornbilll.com" in SPF.
We have added hornbill._domainkey.OURDOMAIN.gov.uk as TXT comment on DNS with DKIM key.

Our workflows are restarting successfully.

I think we're up and running again

[Victor]

@Berto2002 thank you for the update, happy to hear the Direct Outbound solution is suitable and working for you.

[Victor]

We have received information from Microsoft regarding recent changes whereby they have disabled TLS 1.0 and TLS 1.1 on SMTP connectors. However, we are not affected by this particular change as Hornbill already supports TLS up to 1.3. We are continuing to investigate the issue and possible options here.

[Victor]

We identified the root cause for the issue and we are working to deploy a change that should resolve the issue (this is currently being tested). We will post more details about this shortly.

[Martyn Houghton]

@Victor

Thanks for the update and for team progressing this.

Cheers

Martyn

[Victor]

We are almost ready to have the fix deployed. Current ETA is to have the fix deployed by 15.00. I will update this post if anything changes.

[Victor]

We have deployed a patch in all live instances that should address the issue with outgoing email. Emails that failed to be send while the issue was ongoing and currently residing in Sent Items folder can be manually resent. Emails that failed to be send while the issue was ongoing and currently residing in Outbox folder will be automatically sent on the next attempt.

[KrisReynolds]

2 minutes ago, Victor said:

We have deployed a patch in all live instances that should address the issue with outgoing email. Emails that failed to be send while the issue was ongoing and currently residing in Sent Items folder can be manually resent. Emails that failed to be send while the issue was ongoing and currently residing in Outbox folder will be automatically sent on the next attempt.

I might be missing it, but I can't see any resend buttons in the mailbox... is there one hiding somewhere, or do we need to go back into each case and resend that way?

[Martyn Houghton]

@KrisReynolds

It a message at a time from the Outbox folder.

Cheers

Martyn