Jump to content

Critical incident affecting the service in customer instances

Message added by Victor,

The issue has been fixed and service availability fully restored.

We are experiencing an issue with firewall configuration in our data centre. This issue is affecting access to the service. 
Our infrastructure engineers are working on this. Please follow our Hornbill status page for further details here: https://status.hornbill.com/

Recommended Posts

  • Victor unfeatured and unpinned this topic

@Trevor Tinsley Below is the RCA for last weeks outage:

"At 16:26 BST on 14th June our monitoring systems detected a loss of connectivity to one of our datacentres (MDH - Maidenhead). This was a temporary loss of communication between the firewalls brought about by a temporary loss of power to a switch during routine maintenance. The power outage should not have caused any issues, but on this occasion, all remaining firewalls incorrectly took on master status. Steps were taken to resolve the issue by demoting secondary firewalls and this was completed at 16:42. The total impacted time was 16 minutes.

A planned update to the firewalls which includes changes to mitigate this situation is already in place and will be scheduled over the coming months.

The chance of the error occurring again is extremely low and we apologise for the inconvenience this will have caused."

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
  • Create New...