Hornbill Down?

[chriscorcoran]

Hi 

 

Are there issues with Hornbill? All users getting the following when logging in 

 

Failed to initialize application FlowCode Exception (com.hornbill.core/flowcode/fc_ops/adminInitaliseData): An unexpected database error occurred while trying to get role information

Please contact you system administrator or click here to try again

Thanks

Chris

[Everton1878]

we're getting the same, started a few minutes ago

[Everton1878]

seems to be working again now, is it working for you @chriscorcoran?

[Keith Stevenson]

All,

Thanks for the posts. We can inform you that our monitoring system detected an issue at 13:05 which effected a subset of instances . The root cause was identified and steps taken to resolve at 13:08, however it was then also found that this required a restart of 1 of the backened servers which took approximately 5 mins to restart.  All services were fully restored at 13:18.

 

We are still investigating the root cause and will provide a further update with analysis and steps we will undertake to ensure issue cannot happen again shortly.

 

Kind regards

[chriscorcoran]

Yes just come back, thanks

 

[Gary@ADL]

is there an area we can check for server issues? like a status page somewhere? I suppose in a similar fashion that we would mark a service as being effected on our customer support portal for users, only in this case we are the users...  

[Victor]

@Gary@ADL -> https://www.hornbill.com/request/

Type in your Instance ID and email address and you will receive a health check status of your instance.

[Steve Giller]

https://www.hornbill.com/request/

if you enter your instance name and email it will check the status.

[edit] @Victor types more quickly than I do!

[Gary@ADL]

thanks victor  

 

[Ralf Peters]

still geting some Errors popping up randomly  ?? !!

 

[Ralf Peters]

[Guest]

@Ralf Peters Let me look into this for you and get back to you

 

Regards

Pam

[Guest]

@Ralf Peters As we discussed, this was due to the fact that there was a service which needed restarting for your instance. Thanks for confirming that the errors are not coming up anymore. 

Regards

 

Pamela

[Ralf Peters]

@Pamela , it is hanging again   , red error box  "unable to query result- could not connect to server  error code -1"

 

[Keith Stevenson]

All,

We have just witnessed the issue again (Same subset of customers). This occurrence of the issue has now been resolved. The root cause is that a set of temporary resources was exhausted causing a bottleneck whilst other requests waited. The resolve was to clear the block.

This issue has become apparent due to a recent addition to the Service Manager application and advance analytics, which allows the creation of free form reports\joins. If these are not carefully crafted they can create unexpected large volume sets.

We are working to identify any instance that may suffer from this issue and will be contacting the primary\secondary contacts for each instance to provide guidance on report creation whilst we investigate a more permanent solution.

We apologies for any inconvenience.

Kind Regards

Keith Stevenson

 

 

[Keith Stevenson]

All,

Following the above we have now made changes to the Analytics engine and Admin tool used to manage this (Prevents multiple joins that would return an extraordinary number of records). This is currently on our beta testing stream and once tested will be rolled out to all.

We are also changing the way services handle large data sets (In excess of 30GB) and adding additional self correcting mechanisms to avoid issues (These will be rolled out as part of our continuous delivery of the weeks ahead) . Looking even further ahead we already have plans to overhaul reporting that will drastically reduce the impact of any large data sets.

Kind Regards

Keith Stevenson