Critical incident affecting the service in customer instances

[Victor]

Please follow https://status.hornbill.com/. More details to follow.

[Victor]

We are experiencing an issue with firewall configuration in our data centre. This issue is affecting access to the service. Our infrastructure engineers are working on this.

[Deen]

The issue should now be corrected.  Please accept our apologies and let us know if any issues persist.

[Adam Toms]

Thanks @Deen we were also impacted by the issue. Do we have a root cause for this issue as of yet? 

 

Many Thanks

Adam

[Deen]

@Adam Toms I should have an RCA early tomorrow for you.

[Trevor Tinsley]

HI Deen,

Do you have a link to the RCA for Monday's firewall issue?

Thanks.

Trevor.

[Deen]

@Trevor Tinsley I will have that for you shortly and will let you know here when it is ready.

[Deen]

@Trevor Tinsley Below is the RCA for last weeks outage:

"At 16:26 BST on 14th June our monitoring systems detected a loss of connectivity to one of our datacentres (MDH - Maidenhead). This was a temporary loss of communication between the firewalls brought about by a temporary loss of power to a switch during routine maintenance. The power outage should not have caused any issues, but on this occasion, all remaining firewalls incorrectly took on master status. Steps were taken to resolve the issue by demoting secondary firewalls and this was completed at 16:42. The total impacted time was 16 minutes.

A planned update to the firewalls which includes changes to mitigate this situation is already in place and will be scheduled over the coming months.

The chance of the error occurring again is extremely low and we apologise for the inconvenience this will have caused."