slow performance Slow Performance

[Jeremy]

We are expericing a lot of slow performance this morning including people not being able to login or open/log requests.

Is this confined to us?

[Victor]

@Jeremy yes and no. The node for your and @Andrew Tasker instance is experiencing some issues that our infrastructure team is looking into. Will post more updates soon.

[Victor]

@Jeremy all should be working ok now. @davidrb84 FYI

[Jeremy]

@Victor thank you, would it be possible to let us know what happened?

[Victor]

@Jeremy will post an update as soon as our infrastructure team has conducted the investigation

[Keith Stevenson]

Dear @Jeremy

This appears to be caused by 1 of your Accounts (SD AUTO BOT) running a given API (entityAttachDoesExist) and cycling through all file attachments . Its calling this 100s of times a second.  Not sure why or what you are trying to achieve but stopping this would solve your issue

 

Kind Regards

 

Keith Stevenson

[Jeremy]

we will take a look

[Keith Stevenson]

@Jeremy
Seems to have started again in last 2 mins.  This will continue to cause slowness and then eventually tie up the Instance 

KInd Regards

 

Keith Stevenson

 

[Jeremy]

@Keith Stevenson we cannot see that our account is running the API (entityAttachDoesExist) in anything that we have done.

This has happened since the update last night, I am only clutching at straws but did something change API wise in the latest update?

[Stephen Christopher]

We don't ever directly call this API method (entityAttachDoesExist) and everything has been working fine since this morning. :-/

[Keith Stevenson]

@Jeremy
What is the SD Auto Bot account used for? What scripts does it fire.  Also your instance has just got to the critical point so will just have to give it a restart. This will take about 1 minute. 

Kind Regards

Keith Stevenson

[Stephen Christopher]

sdautobot is the account we use for automating some things e.g. moving multiple emails to another folder, importing staff accounts, auto logging jobs via emails, etc.

I've also bounced our main server that fires these scripts.

[Stephen Christopher]

Our server is back up now - i will keep an eye on what scripts are run.

Our tidy inbox script has just ran at 11am (this moves emails from the inbox to another folder) and that seems to have not caused any issues.

[Keith Stevenson]

@Stephen Christopher
At the moment it appears to have calmed down and we will continue to monitor. Im working back through the logs to see what the first occuruence of the API spam was but it created so much noise that cant see wood for trees. 

Kind Regards

Keith Stevenson

[Stephen Christopher]

@Keith Stevenson

Thank you, i understand. Something may have happened yesterday (approx 15:30-16:30) when we decided to move over 2000 emails from one folder to another?

[Keith Stevenson]

@Stephen Christopher
I will have to have a look at yesterdays logs. As for this mornings it started at 07:42:12z and the offending check seems to be on 

h_name = 'image_1.png' 

and then increments  the image check by 1 each time for the call reference IN01007239. It does this close to 160K times in 1 hour (seems to loop ) before the server just starts using more and more RAM and slowing down.  Will need to check if its the same offending callref in the next 2 occurences (after the restarts) but this may give you something to look at 

Kind Regards

Keith Stevenson

 

[Keith Stevenson]

@Stephen Christopher

Its also started again at 11:16 

The times that its fired are 08:42, 09:12, 09:42, 10:12 , 11:12 ( This looks like something scheduled every 30 mins) 

Kind Regards
 

[Stephen Christopher]

@Keith Stevenson

That's our email autologger script - i've been running it for over week without any issues.....strange why it's started behaving in this way.

The API commands i use are:

$mc->invoke("apps/com.hornbill.servicemanager/Incidents", "logIncident");

$mc->invoke("apps/com.hornbill.servicemanager/Requests", "update");

$mc->invoke("apps/com.hornbill.servicemanager/RelationshipEntities", "add");    

$mc->invoke("mail", "deleteMessage");

I'll try and grab the parameters i use in LogIncident...
 

 

 

[Jeremy]

@Keith Stevenson scripts have been stopped but it appears that we have stalled again

[Keith Stevenson]

@Jeremy

Strange I could still see plenty of activity (it had risen but was still OK).. Now restarted

Kind Regards

 

[Keith Stevenson]

@Stephen Christopher

Given it was OK yesterday (After the ESP release) but had issues this morning (After the Service Manager release) I wonder if something has changed in Flowcode. If you can DM me the script I can drop it on a test box and see if we can replicate. 

Kind Regards

 

[Jeremy]

we have also been notified that a request was raised but it spawned a lot of clones....this was raised via self service customer got as far as attaching a quote then spawned lots of requests.

[Stephen Christopher]

@Keith Stevenson

Thanks Keith, i'll message it to you now.