Victor Posted July 2, 2019 Posted July 2, 2019 Due to issues with Cloudflare (Our DNS provider and load balancer) customers may be unable to access live.hornbill.com and their instance. We are working with Cloudflare to investigate and resolve this as a priority Cloudflare Status Investigating - Cloudflare is observing network performance issues. Customers may be experiencing 502 errors while accessing sites on Cloudflare. Please follow: https://status.hornbill.com/We are currently bypassing Cloudflare until the situation is fully resolved. As such all Hornbill instances should be accessible and service operational. UPDATE: Cloudflare fixed the issue at their end and we have reenabled Cloudflare on customer instances last night. All services should be up and running and Hornbill service fully operational.
Victor Posted July 2, 2019 Author Posted July 2, 2019 Cloudflare are working to have the issue fixed and we are liaising with them to work it through. For the time being we disabled the caching mechanism relying on Cloudflare until the situation there is fully resolved. The issue is also posted here: https://www.cloudflarestatus.com/
Victor Posted July 2, 2019 Author Posted July 2, 2019 The issue should not affect Hornbill instances anymore, please let us know if you experience any further issues.
Victor Posted July 2, 2019 Author Posted July 2, 2019 Updates from Cloudflare: Cloudflare outage caused by bad software deploy This is a short placeholder blog and will be replaced with a full post-mortem and disclosure of what happened today. For about 30 minutes today, visitors to Cloudflare sites received 502 errors caused by a massive spike in CPU utilization on our network. This CPU spike was caused by a bad software deploy that was rolled back. Once rolled back the service returned to normal operation and all domains using Cloudflare returned to normal traffic levels. This was not an attack (as some have speculated) and we are incredibly sorry that this incident occurred. Internal teams are meeting as I write performing a full post-mortem to understand how this occurred and how we prevent this from ever occurring again. https://blog.cloudflare.com/cloudflare-outage/
Victor Posted July 3, 2019 Author Posted July 3, 2019 As stated on the Cloudflare blog post, they have resolved the issue and conducted an RCA. The outage was caused by a software misconfiguration deployed in the Cloudflare infrastructure. This particular misconfiguration effectively created a DDoS situation which brought down a large portion of their infrastructure affecting a large number of online services (the rough estimate is that over 16 million websites and online services experienced an outage). The disruption in Cloudflare infrastructure lasted for approximately 25 minutes. Hornbill deactivated the Cloudflare caching as soon as we were aware of their issue to restore the service in customer instances. We are deeply sorry for the disruption and we assure that we are looking to implement a set of measures allowing us to restore the service more rapidly if such a scenario reoccurs. If you are in need of assistance with any fallout issues please let us know. 1
Recommended Posts