Hornbill Development have now pushed out a fix, customer instances should now be back online.
We apologise for the disruption this will have caused. Below is the RCA for the outage:
On Tuesday 5th November an outage affecting ALL instances in all DCs occurred, starting at 1511. The root cause was a planned change to a global settings parameter within our configuration system. This change when pushed caused an internal server error due to how this information was cached, resulting in an error on Login or Opening of new tab informing the user of a "Framework Initialization error"
The change was known and therefore actions to roll back started quickly, however the Internal Server issue with caching caused unexpected results which required further action, including a full restart of all instances.
This was completed at 1620
We have already identified the code in the ESP Platform that caused the caching issue (and therefore the outage) and changes will be made to ensure that this cannot happen again.
We are also reviewing the way in which pushing of global configuration changes are made. These changes happen every 2 minutes when needed , however we will look to restrict them to Out of hours and targeted streams.
We apologise for any inconvenience
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now