Outbound email failures

[Berto2002]

Will you be publishing the cause?

[Victor]

@KrisReynolds there is no "resend all" button I'm afraid. Each email has to be resent, as suggested by Martyn.

[Victor]

In terms of the issue itself, we do have a decent idea of how it came to be, the what's and why's. However, we would like to investigate this a bit further, in bit more detail, before we come back with a more in depth RCA. I will update the thread when this is ready.

[Oscar Stankard]

For anyone that also hadn't seen this, if you click on the envelope icon next to a recipient's name in the email TO field it will give you the option to "Resend Message to Recipient".

 

It's also got a great SMTP session log under the Delivery Status option that's handy.

 

[Berto2002]

Ah, found them and resent ok; 27 failed for us with the earliest being 08.38 on 23/11/22

[Victor]

17 minutes ago, Berto2002 said:

Will you be publishing the cause?

@Berto2002 certainly will, just need to make sure we have all the bits and pieces beforehand.

[Oscar Stankard]

Hi there,

 

Thanks for your efforts getting this sorted, can I ask how the change has gone and what the delivery stats for smart host delivery to Exchange Online are looking like now?

 

Kind regards,

Oscar

[Victor]

@Oscar Stankard we are working on to put everything together, I believe all will become more clear once that is published (soonTM)

EDIT: the email delivery is fine from what we can see, no further issues.

[Gerry]

Following on from the above errors and by way of an explanation. 

Over the last few days/weeks, Microsoft have been updating their servers which changed the priority of TLS version negotiation as part of the key exchange process, and our server was not handling the case where the negotiation was declared as TLS1.3 and that key exchange negotiation failed, there was no fall-back to TLS1.2 which there should have been. So no fall-back and a failing TLS1.3 key exchange negotiation. This was a problem in Hornbill's code, triggered by changes being made by Microsoft. Our initial thought was this was a Microsoft issue, given that our other independent test points on systems like GMail, Postfix and others were all fine. 

A work-around was applied to our code and provided as a hot-fix, this hot fix forced the negotiation down to TLS1.2. We since applied an update to one of the third party libraries we use after their feedback has confirmed the problem is now resolved without the forced downgrade to TLS1.2, that will be in the next platform update.

We apologise for any inconvenience caused, and we appreciate just how important Hornbill is to our customers day to day operations. Unfortunately when working with 3rd-party systems, changes do happen and from time to time that are not within our control and things can get broken. Hopefully we continue to demonstrate that we are able to respond to these kind of events and resolve the issues in a timely manor. 

And for those that want some more technical details, this is the technical bit...
-------
The issue related to TLS 1.3 ClientHello w/ the "key_share" extension. The initial ClientHello message can include pre-generated key shares. While it is not computationally intensive to generate an x25519 key share, and virtually all servers support x25519, it is far more computationally expensive to generate an secp256r1 key share. If our server generated it, then for applications that requires high-performance w/ many connections, the extra work of generating the secp256r1 key share would be significant. Therefore, Hornbill SMTP implementation just pre-generates the x25519 key. If that key is not supported, the server will simply respond with a HelloRetryRequest, in which case Hornbill sends a new ClientHello with the secp256r1 key_share, or whatever other key shares are needed.

Microsoft at some point seems to have stopped supporting x25519, or maybe changed something with the HelloRetryRequest request, such that the HelloRetryRequest failed after a failed negotiation attempt for a TLS1.3 negotiation, its not clear exactly what has changed or why as this is not documented. 

Hornbill SMTP implementation now pre-generates the secp256r1 key share specifically for Microsoft servers.  This is the best generic solution because TLS connections will have better performance for the vast majority of SMTP servers currently supporting x25519
-------