Hornbill service disruption

[Euan Coleman]

Hi Guys,

 

Any system error reports this morning?

 

A lot of the guys in the office here are just getting the loading image but nothing is happening. This issue remains with a full browser restart.

[LifeOfJonny]

Still the same incognito doesn't help. All our users have the problem.

[Keith Stevenson]

We have pushed a change. Can anyone confirm the issue is now working as expected. 

Kind Regards
 

[Everton1878]

same issue here, mine came up saying I had lost connection and to reauthenticate but came up with the loading page

some of my colleagues are still logged in and still able to work

[Mark (ESC)]

Same for us, spinning Hornbill icon of doom .  Time for cup of tea number two!

Clearing cache did not help, we use Edge.

[Victor]

Consolidating threads

[Berto2002]

@Keith Stevenson

Network tab is blank when pressing F12 on the page where we have the loading

[Keith Stevenson]

@Berto2002
Can you click the Disable Cache option and try again

Kind Regards

 

[Martyn Houghton]

5 minutes ago, Keith Stevenson said:

We have pushed a change. Can anyone confirm the issue is now working as expected. 

@Keith Stevenson

We appear to be back working.

Cheeers

Martyn

[P Morris]

I also had to clear the Media Foundation stuff too as well as cache and cookies.

Restarted browser and seems to work fine now

[Berto2002]

SOME USERS ARE ABLE TO WORK AFTER CLEAR Cache.

But most are not. I would say 30:70 ratio we're seeing.

[Kelvin]

Working for us now

[Berto2002]

Disable cache option and reload WORKED for me @Keith Stevenson

but not sure if that is because it's generally working now (as above some reports) or my action there

[Sana]

Clearing browsing history seems to have worked for us - thanks!

[PeterL]

9 minutes ago, Keith Stevenson said:

We have pushed a change. Can anyone confirm the issue is now working as expected. 

Kind Regards
 

Takes a few seconds to load pages but is now loading them at least. 

[Michael Taylor]

Disabling Cache in Dev Tools (F12), Network tab, seems to work as a work-around 

[WFMKC]

All good for us now.  Thanks for the swift response

[Everton1878]

back up and running for us too 

[Berto2002]

We're all up and running

[HGrigsby]

For us too - thank you! 

[Mark (ESC)]

23 minutes ago, Mark (ESC) said:

Same for us, spinning Hornbill icon of doom .  Time for cup of tea number two!

Clearing cache did not help, we use Edge.

We are back, all users.  Thank you.

[Blowerl]

It all working for us. Thanks.

[Keith Stevenson]

Good to hear. Everyone should be back and running. 

KInd Regards
 

[Deen]

For anyone affected, below is the root cause analysis for this disruption:

On the morning of the 29/11 at approximately 09:15 a number of customers began reporting that their Hornbill instances were unavailable.  

After investigating the issue it was determined that there was a problem with the deployment of the latest Core UI framework release that caused some files from the previous version to be served along side the new release, this then lead to the old files being incorrectly cached in Cloudflare alongside files in the new release.  An incompatablity between these releases lead to errors when trying to load the framework.
  
To fix this we deployed a patch by 09:40 to break the cache again so that all requests would get the correct version. We are planning to make changes to the release process to prevent this happening again which will include purging the Cloudflare cache after deployment.

We apologise for the inconvenience this will no doubt have caused.