Force a retry of queued emails

[davidrb84]

Unfortunatly we recently had an issue where the primary hornbill IP was blocked (inbound) for approx 24 hours.

This caused a few issues including blocking email sending (we send mails through our internal SMTP server).

This was found and sorted, however the backlog of emails took ~12 hours to clear.

I assume this is because these mails had tried and failed previously and so had backed off to only retrying after 12 hours or so.

Request: It would be occasionally useful to be able to force an email to retry. Either globally from the admin screens, or from the mailbox interface.

[Alberto M]

Last week we have changed the Hornbill outbound email to use smtp.office365.com. After some days working without issues, today we began to have non delivered emails due to the following error:
--sendCmdToSmtp
readSmtpResponse:
SmtpCmdResp: 554 5.2.0 STOREDRV.Submission.Exception:SubmissionQuotaExceededException; Failed to process message due to a permanent exception with message The message can't be submitted because the sender's submission quota was exceeded. SubmissionQuotaExceededException: The message can't be submitted because the sender's submission quota was exceeded. [Hostname=.(snipped out)..outlook.com]
--readSmtpResponse
 

Meanwhile we've rolled back the outbound email configuration to use the previous configuration with our internal server as a workaround while this problem is not solved and we are trying to get support from Microsoft about this.
As per the Microsoft documentation here https://docs.microsoft.com/en-us/exchange/troubleshoot/send-emails/smtp-submission-improvements “In case any application or device is throttled, the application or device should be designed to retry submitting messages to make sure that the messages get sent.”

It would be nice that Hornbill could have a setting or a feature that allows the emails to be forced to retry in some situations.

We've lost several hours of missed sent emails with this.

[Alberto M]

*URGENT*

We'll need to set our outbound email to office 365 by June 13th as we are still using our expiring outbound email relay server.

Is this something Hornbill could add? A setting or a feature that allows the emails to be forced to retry in some situations (like this exception).

We are trying to arrange a limit override with MS, but this setting would avoid Hornbill customers with higher email traffic to do this.

 

[Alberto M]

Is there anyone that can respond to this?!

 

[Victor]

@Alberto M there are the following system settings that can be helpful in your scenario:

Also, as a Premier Success customer you have the option to expedite a forum post answer/reply by raising a support request in this regard. For future reference  

[Alberto M]

11 minutes ago, Victor said:

Also, as a Premier Success customer you have the option to expedite a forum post answer/reply by raising a support request in this regard. For future reference

Done it meanwhile, thanks 

[Alberto M]

@Victor, thanks for the suggestion, but we have those settings as you show here and we had a lot - really a lot! - of emails not being sent when this issue appeared after we tried to set O365 as outbound email server.

So, apparently this way is not solving this issue.

[Alberto M]

Should I set this to a higher amount?

 

[Victor]

@Alberto M the values shown in my screenshot are the default values for this settings... if having this values you have the issue you described then adjust the values... that's what I meant...

For example... work with the retry timeplan...  like increase the gap for the retry sequences... or add more ... perhaps increase the failure retry time

This being said, however, based on the error you posted, the issue lays with the O365 account having a submission quota. The retries won't be of much help, you would still have the same amount of emails and they will keep hitting the max quota... this really needs to be sorted on O365 side...

[Victor]

Also re this from https://docs.microsoft.com/en-us/exchange/troubleshoot/send-emails/smtp-submission-improvements

In case any application or device is throttled, the application or device should be designed to retry submitting messages to make sure that the messages get sent.

Hornbill is designed to retry submitting messages. It always had this mechanism. The retryTimeplan system setting has 9 values by default which means, by default, if a message fails to be sent, Hornbill will retry a total of 9 times (at the specified intervals) before marking the email as failed to be sent. So the mechanism described in the MS doc exists.

[Alberto M]

@Victor

As this seems not to work with O365, we believe Hornbill should think about a setting to manage this. The error described here:
https://docs.microsoft.com/en-us/exchange/troubleshoot/send-emails/smtp-submission-improvements

"554 5.2.0 STOREDRV.Submission.Exception:SubmissionQuotaExceededException
This change for concurrent connections will better protect the service from large bursts of email messages that are sent by automated systems within a short time period."

We believe that MS O365 is being widely used worldwide and email services that use it should be prepared to avoid "large bursts of email messages that are sent by automated systems within a short time period". If Hornbill is interested in his large customers to not have this problem, this feature should exist in Hornbill.

 

[Alberto M]

@Victor, so, it seems that, when there is a large number of outbound messages retrying one after another, it generates "a large burst within a short time period", even if you tweak the current existing setting in Hornbill.

[jeffreysmith]

@Alberto M office365 will give you this error message if you break one of the following two limits:

• 30 messages per minute
• Recipient rate limit of 10,000 recipients per day

https://docs.microsoft.com/en-us/office365/servicedescriptions/exchange-online-service-description/exchange-online-limits#receiving-and-sending-limits
 

The bottleneck here is office 365 and unfortunately as we relay mail through them for you there is nothing that hornbill can do to bend this limit.

What exactly is causing these big spikes? If it's some kind of monitoring system that sends a flurry of emails you could move this to send from another mailbox rather then office 365.

[Alberto M]

@jeffreysmith, thanks for the explanation.

After a quick investigation in our sent messages folder for this mailbox, it seems that, from time to time, we break this threshold, which should be normal for the thousands of users we have.

 

[Alberto M]

@jeffreysmith, @Victor,

FYI, the answer from MS support when we asked them about changing the emails per minute threshold in O365:

"Thank you for answering the call today, as per the update from our engineering team we cannot increase the rate limit. We can only increase daily send limit as one time activity if required as of now.
Engineering team further said that If you will submit messages too quickly and the send-as is denied this will count against RRL, They suggest to throttle sends to mitigate the issue."

We changed a few settings in the mail.outbound settings and we will be changing our outbound configuration today although we are not convinced this will solve the issue if we hit again the 30 emails per minute threshold.

 

[Alberto M]

@jeffreysmith, @Victor, meanwhile, regarding the "Test connection" button when setting up the outbound settings, with the current button feature and O365 it's impossible to test because Outlook doesn't provide to send using different from address other than your username to log in, and this test is made using a sender do-not-reply@live.hornbill.com 

This is something that should be changed in this button feature.

 

[Alberto M]

@jeffreysmith, @Victor,

Meanwhile, we've set O365 in the outbound email configuration - again - last Monday. Yesterday morning I already had analysts complaining that emails were not being sent. I've tested myself sending an email from a request and it failed immediately,  I mean, 30 seconds after sending it, the email message was already in the sent folder with "failed" status, which apparently means that no mail.outbound retry settings have been applied and they seem useless for this situation.

I then rolled back the outbound email configuration to the previous one and realized that around 200 emails have been rejected yesterday morning due to this.

Please check what can be done to help solving this.

Regards,

Alberto

(this is our mail.outbound retry settings)