Jump to content

Hornbill Down?


chriscorcoran

Recommended Posts

Hi 

 

Are there issues with Hornbill? All users getting the following when logging in 

 

Failed to initialize application FlowCode Exception (com.hornbill.core/flowcode/fc_ops/adminInitaliseData): An unexpected database error occurred while trying to get role information

Please contact you system administrator or click here to try again

Thanks

Chris

  • Like 1
Link to comment
Share on other sites

All,

Thanks for the posts. We can inform you that our monitoring system detected an issue at 13:05 which effected a subset of instances . The root cause was identified and steps taken to resolve at 13:08, however it was then also found that this required a restart of 1 of the backened servers which took approximately 5 mins to restart.  All services were fully restored at 13:18.

 

We are still investigating the root cause and will provide a further update with analysis and steps we will undertake to ensure issue cannot happen again shortly.

 

Kind regards

  • Like 1
Link to comment
Share on other sites

is there an area we can check for server issues? like a status page somewhere? I suppose in a similar fashion that we would mark a service as being effected on our customer support portal for users, only in this case we are the users... :P 

Link to comment
Share on other sites

@Ralf Peters As we discussed, this was due to the fact that there was a service which needed restarting for your instance. Thanks for confirming that the errors are not coming up anymore. 

Regards

 

Pamela

Link to comment
Share on other sites

All,

We have just witnessed the issue again (Same subset of customers). This occurrence of the issue has now been resolved. The root cause is that a set of temporary resources was exhausted causing a bottleneck whilst other requests waited. The resolve was to clear the block.

This issue has become apparent due to a recent addition to the Service Manager application and advance analytics, which allows the creation of free form reports\joins. If these are not carefully crafted they can create unexpected large volume sets.

We are working to identify any instance that may suffer from this issue and will be contacting the primary\secondary contacts for each instance to provide guidance on report creation whilst we investigate a more permanent solution.

We apologies for any inconvenience.

Kind Regards

Keith Stevenson

 

 

  • Like 1
Link to comment
Share on other sites

All,

Following the above we have now made changes to the Analytics engine and Admin tool used to manage this (Prevents multiple joins that would return an extraordinary number of records). This is currently on our beta testing stream and once tested will be rolled out to all.

We are also changing the way services handle large data sets (In excess of 30GB) and adding additional self correcting mechanisms to avoid issues (These will be rolled out as part of our continuous delivery of the weeks ahead) . Looking even further ahead we already have plans to overhaul reporting that will drastically reduce the impact of any large data sets.

Kind Regards

Keith Stevenson

  • Like 1
Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...