Hornbill Performance issues and slowdowns

[samwoo]

Hello,

We are having performance issues and have been advised by Hornbill Support to ask on here.

Most of our users have been reporting that Hornbill is inconsistently slow, almost everyone has complained that on random occasions Hornbill slows down to a crawl… before picking up speed again after some time. Some users tend to switch between IE or Chrome if one becomes slow… basically opening a new instance of Hornbill each time.

This happens in both Chrome and in Internet Explorer.

Other web applications are fine however. There are no network issues, system resource usages on the users machines are low, other applications such as Outlook are working fine when Hornbill slows down as does google and other web apps.

The slow downs appear to be related/occurring mostly within the Service Manager.

Here is a comment from one of the most active users of Hornbill (a Service Desk Analyst)

Essentially the best situation to notice the slowdown with is when logging a request and clicking the “view” option at the end of the log process. Occasionally, this takes a long time to come up with the details of the ticket but most of the time this is rather quick as expected.

I think that this is possibly due to more users being connected to a Hornbill server than there were before? Or possibly because more than 1/2/3 users performing an action within Hornbill at the same time?

One time 24 requests were closed off at almost the same time (less than a minute apart) and this slowed the system to a crawl for everyone within our department who uses Hornbill, some people could not even login at all with lots of server errors occurring.

It took over 2 minutes for one call to be resolved after clicking the resolve button. But other web apps and internet was working fine with no slow downs at all.

It's been going on for some time now... the first I heard of it was about 3 weeks ago, and it's still going on today and people are getting frustrated.

Can someone advise? I don't log calls often so I haven't noticed it from my end, but I've witness other users instance of Hornbill slowing down when Outlook is the only other application open and was working fine.

Thanks,

Samuel

ps. Is it possible for there to be another Forum for more General queries such as this one? I put it in the Service Manager area because that is where most of the slowdowns are experienced (as this is the area most used), there have been instances as above where approx 24 calls being closed at the same time caused issues for users trying to log in to Hornbill.

[nasimg]

Have you been able to confirm the slow downs from an external laptop/ home pc - I say this because in our thin client environment Service Manager is slower than if I use my personal laptop. I put this down to our infrastructure.

[Gerry]

Hi Samuel,

We are going to take a look at the logs on your instance and monitor more closely for a few days to see if we can understand what you are seeing. An at-a-glance look does not show anything untoward apart from some spikes in use where response times on some API calls take a little longer than usual. In order to understand whats happening we need to study use patterns for a few days so please bear with us. If there is anything we can do short-term we will of course do that. Either way we will post back here with an update soon.

Gerry

[Gerry]

Hi Samuel,

Just a quick update, we now have three teams looking at performance at three different layers in our stack, this is mostly and observe and hypothasize exercise so I have nothing more to feedback at this time apart from letting you know that we are taking your feedback and seriously, and are looking at every aspect of it in order for us to understand what you are seeing. We can see some areas of our system where there is opportunity to optimise but as of now there is no single thing that stands out as a culprit, where we do see spikes, it just seems to be general load related - that though is only an initial impression, we still have a lot more to consider.

If you would be good enough to relay to your users that we are looking at our systems and the response times so would appreciate it if you can bear with us for a few days while we make our observations.

Gerry

[samwoo]

Hi Gerry,

Many thanks for responding. i have relayed this to our users. We really appreciate that the issue is being looked into.

Thanks,

Samuel.

[Gerry]

Hi Samuel,

Me and the team here at Hornbill take the quality of the service we provide very seriously so no thanks required, you should expect nothing less than a first class service from us. It is not always easy and we don't always get things 100% right, but we will always solve problems we can see and find new ways of making what we provide better. I will post with an update in a weeks time when I am sure we will know a lot more about what you have been seeing - thanks for your patience.

Gerry

[Gerry]

Hi Samuel,

We have been monitoring your instance and observed a number of areas where our application could be made more efficient. To the best of our present determination at this time, its not one specific issues, its a combination of smaller performance issues that are culminating to sometimes show slow responses to API calls around call request management. This only becomes apparent when there are spikes of busy time which is why its been unpredictable.  We believe there are a number of areas we can improve and so are planning how to deploy these changes as we make them, so you can expect to see incremental improvements in performance when logging a request over the next 3 to 4 weeks. Here are some things we know for sure. 

• We found that our SM implementation is using Flowcode security elevations, in some cases 15 or more in a single call. We found that our implementation in the core was less than efficient in the way it handled these elevations.  We have already applied a fix for this which improves that performance, this will be in production very soon (if not already), although in isolation thats only a small overall improvement, but an important one one the less.
• We know that in some operations (in particular logNewRequest) where we have duplicate redundant queries. These have come about as we have expanded functionality and have modularised our code to handle the functionality - we plan to re-structure the flow of code in order to remove redundant duplicate queries.
• In some areas we have been using SELECT * FROM .... on tables that have a large number of columns, only to then use a couple of values. This is a bandwidth and resource hog on our database layer and needs to be optimised. 
• We are doing far too much of the functional logic during synchronous  operation of raising a new call, which has the effect of appearing much slower than it should be.  We will be re-factoring some of the code so much more of this is done asynchronously after the operation completes. 

So we have lots of places to improve and optimise and we are on the case.  It is inevitable that we will have to go through refactoring and optimisation cycles like this from time to time, but as I said before we do take this stuff very seriously, the last think we want is any of our users having the perception that our service is slow or clunky - we ideally want our users telling their friends and colleges how awesome Hornbill is

We will keep this post updated as we make further progress, I have asked SM application team to post an update here this week.  In the meantime, if you have any other relevant information please do post it here.  Likewise,  if you see improvements please also post here

Thanks

Gerry

[Gerry]

Nasim,

We are looking at putting our services through a much better front end cache, we are doing this for redundancy as well as initial load performance for users that are out-of-region.  However, this might offer you marginal improvements when remote as the cache provider has much bigger backbone links than we do so static cacheable content (the UI basically) should get served a bit faster than it does currently. 

Gerry

[Ryan]

The SM team have been implementing some of the optmisations outlined above to the logRequest process this week. The result is a noticeable increase in performance, with other methods still being investigated.

The team will be looking at releasing the improvements made next week, and applying what was learned during this process to other existing areas of the SM application over the coming weeks and months.

Ryan

[gwynne]

Hello

I would like to note that we are also suffering speed issues I must admit they don't seem to be as bad at them moment so not sure if the optimisations have been pushed out globally, but looking forward to further improvements.

 

[Ryan]

An update on this, the build has not gone out globally as yet, it has been created and is currently going through our testing process.

The improvements will be in Service Manager 2.28 which should be out early this week.

Thanks,

Ryan