Jump to content

Hornbill connectivity issues - 502 Bad Gateway [RESOLVED]


Recommended Posts

Due to issues with Cloudflare (Our DNS provider and load balancer) customers may be unable to access live.hornbill.com and their instance. We are working with Cloudflare to investigate and resolve this as a priority

Cloudflare Status

Investigating - Cloudflare is observing network performance issues. Customers may be experiencing 502 errors while accessing sites on Cloudflare.

Please follow: https://status.hornbill.com/

We are currently bypassing Cloudflare until the situation is fully resolved. As such all Hornbill instances should be accessible and service operational.

UPDATE: Cloudflare fixed the issue at their end and we have reenabled Cloudflare on customer instances last night. All services should be up and running and Hornbill service fully operational. 

Link to comment
Share on other sites

  • Victor pinned and featured this topic

Updates from Cloudflare:

Cloudflare outage caused by bad software deploy

This is a short placeholder blog and will be replaced with a full post-mortem and disclosure of what happened today.

For about 30 minutes today, visitors to Cloudflare sites received 502 errors caused by a massive spike in CPU utilization on our network. This CPU spike was caused by a bad software deploy that was rolled back. Once rolled back the service returned to normal operation and all domains using Cloudflare returned to normal traffic levels.

This was not an attack (as some have speculated) and we are incredibly sorry that this incident occurred. Internal teams are meeting as I write performing a full post-mortem to understand how this occurred and how we prevent this from ever occurring again.


Link to comment
Share on other sites

As stated on the Cloudflare blog post, they have resolved the issue and conducted an RCA. The outage was caused by a software misconfiguration deployed in the Cloudflare infrastructure. This particular misconfiguration effectively created a DDoS situation which brought down a large portion of their infrastructure affecting a large number of online services  (the rough estimate is that over 16 million websites and online services experienced an outage). The disruption in Cloudflare infrastructure lasted for approximately 25 minutes. Hornbill deactivated the Cloudflare caching as soon as we were aware of their issue to restore the service in customer instances. We are deeply sorry for the disruption and we assure that we are looking to implement a set of measures allowing us to restore the service more rapidly if such a scenario reoccurs.

If you are in need of assistance with any fallout issues please let us know.

  • Thanks 1
Link to comment
Share on other sites

  • Victor changed the title to Hornbill connectivity issues - 502 Bad Gateway [RESOLVED]
  • Victor unfeatured, unpinned and locked this topic
This topic is now closed to further replies.
  • Create New...