Unable to access HB service due to an issue with application subscription [RESOLVED]

[Victor]

@Keith we're deploying a fix so issue is fixed instance by instance, might be your instance had the fix deployed but can't confirm as I am waiting for updates from our infrastructure team

[Keith]

Thanks Victor. Hope others are resolved soon too. 

 

[Victor]

Will be. Unfortunately this fix cannot be deployed on all affected instances at once so some will be restored sooner than others...

 

[Alberto M]

7 minutes ago, Keith said:

Seems to be working now.

Here not yet.

I believe this will be solved in a instance by instance basis.

[Deen]

@Alberto M yes, the fix is being separately applied to each instance.

[Ann-MarieHolloway]

1 hour ago, Victor said:

Will be. Unfortunately this fix cannot be deployed on all affected instances at once so some will be restored sooner than others...

 

Hi @Victor do you happen to have a timescale for the roll out of the updates

[Deen]

The fixes are being applied now.  @Ann-MarieHolloway i'll let you know when your instance is updated.

[Ann-MarieHolloway]

Thank you @Deen my apologies, I know it's how long is a piece of string and appreciate your assistance.

[Alberto M]

@Deen, I can see the subscription list populated now. Can you please confirm it's solved for us?

 

[Victor]

@Alberto M almost...

 

[Deen]

@Alberto M should be done now...

[Deen]

@Ann-MarieHolloway you instance should be updated also now..

[Victor]

@all

The fix has now been deployed in most of the affected instances and the issue should be resolved. Please let us know if not or any other issues.

[Alberto M]

@Deen, @Victor,

Thanks a lot!!

 

[George Warren]

1 minute ago, Victor said:

@all

The fix has now been deployed in most of the affected instances and the issue should be resolved. Please let us know if not or any other issues.

Hello - Could we please be added to the queue if not already, my colleague Colin Chalmers did post earlier.
 

Thanks

[Victor]

@George Warren the fix was deployed in your instance. Do you still have issues accessing SM functionality?

[George Warren]

6 minutes ago, Victor said:

@George Warren the fix was deployed in your instance. Do you still have issues accessing SM functionality?

Sorry needed to log out and back in fully. Apologies. 

[Ann-MarieHolloway]

14 minutes ago, Deen said:

@Ann-MarieHolloway you instance should be updated also now..

Yes, all working correctly thanks everyone.

[Deen]

@Ann-MarieHolloway good to hear, we'll be in touch with the root cause shortly.

[Guest Paul Davis]

The RCA for this issue is as follows:

Around 12:30 BST on Wednesday last week, a small number of Hornbill customers were impacted by an issue that left them without access to Hornbill Service Manager for up to three hours. Full service was restored to those customers, and a fix has since been deployed to prevent any further occurrence of this issue. The issue was not caused by any application or platform update, but was the result of a pre-existing problem with our SaaS configuration management system.

The issue was ultimately identified as being caused by incorrect handling of a database communication error for a particular part of our SaaS configuration management system. This led to incorrect subscription values being applied to a small number of instances in our UK data center.

This issue caused significant problems for customers over an extended period of time and we sincerely apologize for the temporary loss of service. Although the number of customers impacted was small and whilst outages of this nature are very rare, we recognize Hornbill fell short of the high standards we set ourselves. A Lesson Learnt Review is always conducted after major incidents of this kind and in this instance, we have already made changes to ensure this could not happen again, and we are working on further improvements that will make this particular area even more robust in the future.

Once again, we are sorry for the inconvenience caused to those affected.