Email Notification Errors in mailbox

[lcottell]

@Victor All good for us also, Victor. Thank you!

[Victor]

@all

The fix for this morning issue has been deployed across all affected instances and full service restored in all instances.

We are deeply sorry for all the troubles this has caused for you. If there are any other issues resulted from this disruption please let us know via the ongoing support requests (if you have one) or let us know here so we can assist with this.

We are also conducting a postmortem analysis of the incident and once this analysis is complete we will come back with the details here. 

[Victor]

@all

On 26/03/2020 we deployed a platform software automatic update in our host controllers which was applied on all live instances overnight. Code changes in this update caused IPC blocking behavior that would cause Hornbill APIs to become unresponsive and backlog. The APIs would eventually time out and fail which resulted in service disruption and outage across multiple customer instances. How this event unfolded: at 07:20 customers reported that their mailbox is receiving email processing failures notifications. Initial investigation revealed there was an issue processing certain emails where Hornbill was unable to process them. Initial measures revolved around fixing this issue and initial measures indicated the issue was resolved. At 08:30 it became apparent the issue was not isolated to the mail service and other areas were also affected (confirmed by reports of various issues with BPE) and the initial measures also did not appear to be effective as the issue resurfaced. Further investigation revealed that for an unknown reason at that time, all HTTP connections were reporting failures, suggesting the issue is affecting the instance services. We deployed a further set of measures by restarting services on all affected instances. Although this resolved the issue, it only worked briefly, the issue resurfaced shortly after the restart which indicated the issue lays somewhere outside the instance services. At 08:45 we located the issue in the logging system managed by the host controller and a decision was made to roll back the host controller update deployed the previous day and applied overnight. We immediately started the rollback on all live nodes and we completed this operation at 10:20 when full service was restored across all customer instances.

While we know the issue lays somewhere in the event logging mechanism we don't have the full details of how the IPC blocking occurred and we are conducting further investigations. As an interim measure, we will not update our host controllers until the issue is fully understood and the code changes to prevent the issue from happening. Further measure includes additional checks in place to ensure the event log mechanism will not cause this or similar issues in the future.

We unreservedly apologise for all the troubles caused by this service outage. If you have any queries about this or if we can be of assistance with anything else please let us know.