Jeremy Posted October 7, 2020 Share Posted October 7, 2020 We are expericing a lot of slow performance this morning including people not being able to login or open/log requests. Is this confined to us? Link to comment Share on other sites More sharing options...
Victor Posted October 7, 2020 Share Posted October 7, 2020 @Jeremy yes and no. The node for your and @Andrew Tasker instance is experiencing some issues that our infrastructure team is looking into. Will post more updates soon. Link to comment Share on other sites More sharing options...
Victor Posted October 7, 2020 Share Posted October 7, 2020 @Jeremy all should be working ok now. @davidrb84 FYI Link to comment Share on other sites More sharing options...
Jeremy Posted October 7, 2020 Author Share Posted October 7, 2020 @Victor thank you, would it be possible to let us know what happened? Link to comment Share on other sites More sharing options...
Victor Posted October 7, 2020 Share Posted October 7, 2020 @Jeremy will post an update as soon as our infrastructure team has conducted the investigation 1 Link to comment Share on other sites More sharing options...
Keith Stevenson Posted October 7, 2020 Share Posted October 7, 2020 Dear @Jeremy This appears to be caused by 1 of your Accounts (SD AUTO BOT) running a given API (entityAttachDoesExist) and cycling through all file attachments . Its calling this 100s of times a second. Not sure why or what you are trying to achieve but stopping this would solve your issue Kind Regards Keith Stevenson Link to comment Share on other sites More sharing options...
Jeremy Posted October 7, 2020 Author Share Posted October 7, 2020 we will take a look Link to comment Share on other sites More sharing options...
Keith Stevenson Posted October 7, 2020 Share Posted October 7, 2020 @Jeremy Seems to have started again in last 2 mins. This will continue to cause slowness and then eventually tie up the Instance KInd Regards Keith Stevenson Link to comment Share on other sites More sharing options...
Jeremy Posted October 7, 2020 Author Share Posted October 7, 2020 @Keith Stevenson we cannot see that our account is running the API (entityAttachDoesExist) in anything that we have done. This has happened since the update last night, I am only clutching at straws but did something change API wise in the latest update? Link to comment Share on other sites More sharing options...
Stephen Christopher Posted October 7, 2020 Share Posted October 7, 2020 We don't ever directly call this API method (entityAttachDoesExist) and everything has been working fine since this morning. :-/ Link to comment Share on other sites More sharing options...
Keith Stevenson Posted October 7, 2020 Share Posted October 7, 2020 @Jeremy What is the SD Auto Bot account used for? What scripts does it fire. Also your instance has just got to the critical point so will just have to give it a restart. This will take about 1 minute. Kind Regards Keith Stevenson Link to comment Share on other sites More sharing options...
Stephen Christopher Posted October 7, 2020 Share Posted October 7, 2020 sdautobot is the account we use for automating some things e.g. moving multiple emails to another folder, importing staff accounts, auto logging jobs via emails, etc. I've also bounced our main server that fires these scripts. Link to comment Share on other sites More sharing options...
Stephen Christopher Posted October 7, 2020 Share Posted October 7, 2020 Our server is back up now - i will keep an eye on what scripts are run. Our tidy inbox script has just ran at 11am (this moves emails from the inbox to another folder) and that seems to have not caused any issues. Link to comment Share on other sites More sharing options...
Keith Stevenson Posted October 7, 2020 Share Posted October 7, 2020 @Stephen Christopher At the moment it appears to have calmed down and we will continue to monitor. Im working back through the logs to see what the first occuruence of the API spam was but it created so much noise that cant see wood for trees. Kind Regards Keith Stevenson Link to comment Share on other sites More sharing options...
Stephen Christopher Posted October 7, 2020 Share Posted October 7, 2020 @Keith Stevenson Thank you, i understand. Something may have happened yesterday (approx 15:30-16:30) when we decided to move over 2000 emails from one folder to another? Link to comment Share on other sites More sharing options...
Keith Stevenson Posted October 7, 2020 Share Posted October 7, 2020 @Stephen Christopher I will have to have a look at yesterdays logs. As for this mornings it started at 07:42:12z and the offending check seems to be on h_name = 'image_1.png' and then increments the image check by 1 each time for the call reference IN01007239. It does this close to 160K times in 1 hour (seems to loop ) before the server just starts using more and more RAM and slowing down. Will need to check if its the same offending callref in the next 2 occurences (after the restarts) but this may give you something to look at Kind Regards Keith Stevenson Link to comment Share on other sites More sharing options...
Keith Stevenson Posted October 7, 2020 Share Posted October 7, 2020 @Stephen Christopher Its also started again at 11:16 The times that its fired are 08:42, 09:12, 09:42, 10:12 , 11:12 ( This looks like something scheduled every 30 mins) Kind Regards Link to comment Share on other sites More sharing options...
Stephen Christopher Posted October 7, 2020 Share Posted October 7, 2020 @Keith Stevenson That's our email autologger script - i've been running it for over week without any issues.....strange why it's started behaving in this way. The API commands i use are: $mc->invoke("apps/com.hornbill.servicemanager/Incidents", "logIncident"); $mc->invoke("apps/com.hornbill.servicemanager/Requests", "update"); $mc->invoke("apps/com.hornbill.servicemanager/RelationshipEntities", "add"); $mc->invoke("mail", "deleteMessage"); I'll try and grab the parameters i use in LogIncident... Link to comment Share on other sites More sharing options...
Jeremy Posted October 7, 2020 Author Share Posted October 7, 2020 @Keith Stevenson scripts have been stopped but it appears that we have stalled again Link to comment Share on other sites More sharing options...
Keith Stevenson Posted October 7, 2020 Share Posted October 7, 2020 @Jeremy Strange I could still see plenty of activity (it had risen but was still OK).. Now restarted Kind Regards Link to comment Share on other sites More sharing options...
Keith Stevenson Posted October 7, 2020 Share Posted October 7, 2020 @Stephen Christopher Given it was OK yesterday (After the ESP release) but had issues this morning (After the Service Manager release) I wonder if something has changed in Flowcode. If you can DM me the script I can drop it on a test box and see if we can replicate. Kind Regards Link to comment Share on other sites More sharing options...
Jeremy Posted October 7, 2020 Author Share Posted October 7, 2020 we have also been notified that a request was raised but it spawned a lot of clones....this was raised via self service customer got as far as attaching a quote then spawned lots of requests. Link to comment Share on other sites More sharing options...
Stephen Christopher Posted October 7, 2020 Share Posted October 7, 2020 @Keith Stevenson Thanks Keith, i'll message it to you now. Link to comment Share on other sites More sharing options...
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now