Jump to content

Hornbill Instance Unavailable


Victor

Recommended Posts

Hi

On the back of this...

Would it be possible to implement a fail over / test instance?

If we had been live with this system and this had happened we would have been unable to access our service desk for over 2 hours before a resolution was found. (which I am sure you are hearing from other customers)

I can only assume that the issue arose due to one of the updates rolled out yesterday (obviously correct me if I am incorrect in my assumptions)

To that end could you not roll out test instances to patch first, wait 72 hours for them to be tested by customers and then (assuming no issues) roll out the patch to live?

Or provide a fail over instance which is up to date as far as just before the patch (ie patch roll out at 0100, fail over instances up to date to 0030)

Being unable to provide updates or carry out work on existing calls for two hours and having to resort to alternative call logging processes and then spending time inputting the data into SM when it is back up isn't really an option for us.

This is the largest of a number of issues over the last couple of weeks which would have made our instance either fully or partially unusable which makes me nervous about moving to a fully hosted solution with no fail over / test instance.

Thanks

Dan

Link to comment
Share on other sites

Hi Dan,

We are still investigating the nature of the problem we encountered so I don't have exact details yet, but I will state that we did in fact test this change (relating to subscription enforcement) very intensely over a number of weeks and did not find any issues.  So a roll-out to any kind of UAT instance might not have found the problem either.  Hornbill is not like on-premise software, its not just software that we run on a server and host for you, its a distributed multi-layer, multi-tenant architecture so we cant just make changes for one customer, it does not work like that.  Each customers data is fully segregated of course but the application and services logic is not.   In this case, only a very small number of live instances (I will report exact numbers and more information when I have the specifics) were actually affected with most customers being unaffected by the updates applied to our platform.

So this brings us back to the nature of the problem and we are investigating why this failed on a small number of instances, its currently not clear to us yet so we are in code review and seeking to establish why our extensive platform test automation did not find this problem, once we know we will have test coverage and it will not happen in the future. I expect we should know everything we need to by the end of play today. 

In terms of your concern in relation to using a cloud service - we do deploy Hornbill on a continuously rolling basis and while we run into the odd problem (this being and obvious example) our track record for uptime and availability is still very high (over 99.95% consistently).  However, if you absolutely and categorically cannot tolerate any kind of outage at all then perhaps a cloud service is not the right choice for your organisation, not only because we as the service provider might suffer a problem which takes a little time to investigate and resolve, but networking between your location and our data centre might fail which is outside of ours and your control.  I often say this but if you judge on the overall service and the value we deliver through our continuous deployment approach, I would hope the minor glitches we encounter would be tolerable. 

I will post a further update later on once I have more information. 

Gerry

Link to comment
Share on other sites

Hi Dan (and all),

We have continued to work on understanding why this problem occurred, although we can see the problem on some instances we have yet to successfully re-create the problem.  The issue relates to the way in which instanced take on board subscription parameters from our Subscription Manager application and we are still working to identify the root cause, and will continue to do so until we have a full and thorough understanding of the problem.  I will post more information when I have it. 

Gerry

Link to comment
Share on other sites

  • Victor unpinned this topic
  • 1 month later...

Hi Gerry 

This is currently affecting only one user,  an analyst is trying to log on and are getting:

"You are allocated rights to the 'com.hornbill.servicemanager' application but your organizations subscription level for this application has been reached. Please contact your system administrator

Please contact you system administrator or click here to try again"

 

Looking at the subscriptions on our instance we are seeing a difference - is there a synching issue with our accounts?

158.JPG

 

Link to comment
Share on other sites

Hi Nasim,

There was an issue we found where the last user was not being correctly allocated so this may well be that problem. An update is rolling out this evening which includes the fix to that so I expect this will be resolved first thing in the morning for you.  If not I will make investigating this a priority for the team and find a workaround. Hope thats ok. 

Gerry

Link to comment
Share on other sites

Thanks - we have noticed a few users who may potentially have this problem, something around the us demoting the user to a basic user and changing back. Let me know if you want to check a few user who I think may get the subs error

"You are allocated rights to the 'com.hornbill.servicemanager' application but your organizations subscription level for this application has been reached. Please contact your system administrator"

Link to comment
Share on other sites

Hi Nasim,

Yes by the sounds of what you are saying, this is definitely one of the issues we have resolved.  There is also another partly related problem that is caused by the LiveChat app, that is also fixed but will not be in live until the end of this week, but that does not appear to be affecting you so I expect tonights update will resolve your issue. 

If you are still seeing the problem tomorrow please let us know and we will get on it right away. 

Thanks,

Gerry

 

Link to comment
Share on other sites

Hi @Gerry 

We have applied 2.36.5 patch, so after getting the errors with 2.36.3 (BPM and logging a incident which wasn't good), I've finally had a chance to test the initial issue with the subscription level error. This is error is still here. We are seeing this on approx. 3 accounts I'm aware of (possibly more) so would welcome any assistance in getting this issue resolved.

subs_error.JPG

Nasim

 

Link to comment
Share on other sites

Hi Nasim,

I don't have access to your system to log in, we are pretty strict about that around here :) I will ask one of the guys to take a look into your logs and see what is occurring. We have verified the above fixes on other instances are working fine so not sure what exactly is going on here.  It looks like you are completely bottomed out on your subscriptions though so perhaps thats the problem.... 

lb_sub.png

We are also going to expand the following view to include the users Full Name, Job Title and Last Logon Time... these columns will be sortable, this should give you what you need. 

lb_sub1.png

Gerry

Link to comment
Share on other sites

Hi @Gerry

Your team do have access to our admin account so you should be able to log in directly. Issue we see is not from added more staff, but existing analysts are not able to log in. We have identified 3 so far (getting the above error), so something is not keeping track of the licences correctly....Although I haven't tested those users against the latest patch.

I look forward to you getting to the bottom of this as we are relying on this subs menu to keep track of the licenses

Regards

Nasim

 

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...