Keith Stevenson

All, This has now been resolved. Please check and confirm. We will provide a full explanation shortly Kind Regards Keith Stevenson

Micheal, Thanks for the reply. Your instance is configured for us to Download email from your internal mail server via IMAP, so we would be connecting\pulling and uploading nothing (I suppose your router may count the commands we use to connect\view IMAP mailbox and download anything it finds as uploads but thats going to be far less than 37.4GB - probably < 10mb for entire day). Can you provide the report as a private message? Perhaps its the wording thats wrong? Kind Regards Keith Stevenson

Micheal, Thanks for the post. Can you confirm where those stats come from? Kind Regards

Prathmesh, Thanks for the confirmation. We noticed at 8:39 that one of our nodes was acting strange and refusing connections. Investigations showed that this was the result of a deadlock on the back end database. The lock was cleared around 8:48 and once done your instance could again connect. We are investigating the root cause and will add additional monitoring to ensure that this will not occur again. Kind Regards Keith Stevenson

All, Following the above we have now made changes to the Analytics engine and Admin tool used to manage this (Prevents multiple joins that would return an extraordinary number of records). This is currently on our beta testing stream and once tested will be rolled out to all. We are also changing the way services handle large data sets (In excess of 30GB) and adding additional self correcting mechanisms to avoid issues (These will be rolled out as part of our continuous delivery of the weeks ahead) . Looking even further ahead we already have plans to overhaul reporting that will drastically reduce the impact of any large data sets. Kind Regards Keith Stevenson

All, We have just witnessed the issue again (Same subset of customers). This occurrence of the issue has now been resolved. The root cause is that a set of temporary resources was exhausted causing a bottleneck whilst other requests waited. The resolve was to clear the block. This issue has become apparent due to a recent addition to the Service Manager application and advance analytics, which allows the creation of free form reports\joins. If these are not carefully crafted they can create unexpected large volume sets. We are working to identify any instance that may suffer from this issue and will be contacting the primary\secondary contacts for each instance to provide guidance on report creation whilst we investigate a more permanent solution. We apologies for any inconvenience. Kind Regards Keith Stevenson

All, Thanks for the posts. We can inform you that our monitoring system detected an issue at 13:05 which effected a subset of instances . The root cause was identified and steps taken to resolve at 13:08, however it was then also found that this required a restart of 1 of the backened servers which took approximately 5 mins to restart. All services were fully restored at 13:18. We are still investigating the root cause and will provide a further update with analysis and steps we will undertake to ensure issue cannot happen again shortly. Kind regards

Alex, Glad to hear that this is resolved. We have now identified the root cause (issue with stale Database connections getting reused rather than creating new ones) and will change our code to ensure that this doesnt happen again. We will also add additional monitoring checks to ensure that if such an event occurred in the future we are notified (and can take action) before it becomes an issue for yourselves. Kind regards Keith Stevenson

Alex, We have cleared the blocked connections. Please try again and confirm that this is now OK. We are still investigating the root cause and will provide an update once we fully understand the issue. Kind Regards Keith Stevenson

Gingib. Thanks for the post. No changes have been made to the SPF record since 9/15/2016 at 10:43:33 AM and having checked the SPF record everything looks as it should. Can you provide the full headers of an email that was sent and blocked so we can see where it got to. Kind Regards Keith Stevenson

Reena, One option that might help whilst we review the possibility of changes would be to make use of a font designed for Dyslexia. This font (available from https://www.dyslexiefont.com/en/dyslexia-font/) is free for home\personal use and in Firefox (or chrome via extension) its a simple task to change all web pages to use this rather than ours. (See https://support.mozilla.org/en-US/kb/change-fonts-and-colors-websites-use) You can also change the background colour to help (Usually to Yellow) which should achieve your requirements. If the above helps im sure we can look at adding more options for accessibility in the future. Kind Regards Keith Stevenson

Everton1878 Sorry to hear that your having trouble. There should be no issue as the DNS record for the above was fully propagated last week and the TTL on any old record should have passed easily by now. Can you do a NSLOOKLUP for service.hornbill.com from the CMD prompt on an effected machine and post the results as this will show us what DNS server your using, the records TTL etc. If you can also confirm this happens in all browsers that would help (Chrome does its own caching of DNS) Kind regards

All, Following on from the above posts and as clarification. We monitor all service endpoints but have been monitoring them via their IP addresses and not using their FQDN (domain name), we also monitor our DNS service provider and to now that has worked ok. We made a decision to transition over to CloudFlare to take advantage of their superior service as well as future global caching to help accelerate our application content, this was executed last night. Unfortunately we misconfigured the DNS entry for service.hornbill.com but our monitoring continued to show the service as available because we were monitoring the service via its IP address and not its FQDN. We have now reviewed our monitoring setup and re-configured to use the domain name rather than IP address for all services to ensure this gap in our monitoring does not happen in the future. Sorry for any inconvenience caused. Kind Regards Keith Stevenson

Adrian, Sorry to hear that your having problems. Ive looked at the server and although your instance was taking a large amount of RAM about 30 minutes ago, it seems to have settled down. From the logs I can see users logged it and performing actions. Can you confirm that you are still having issues? As for the reason why the RAM was high, the logs state that an analyst was processing an attachment with large number of attachments, but I will need our development team to investigate further. Look forward to your response. Kind Regards Keith Stevenson

Ben, If you goto the Admin portal, under Home->System->Email-Templates you can click to edit the required template and then change the variable to mixed case ( ie H_Summary for mixed, h_summary for lower or H_SUMMARY for upper). Kind Regards Keith Stevenson

Ben, Thanks for the question. The case of the variable in the email template dictates the case of the text. Therefore, if your variable is all lower case the email will be all lower case. If you use mixed case for the variable namethen it will respect the case as it is in the database. Kind Regards Keith Stevenson

All, This should now be fixed in Live. (It was actually fixed late Friday evening but only just saw this post and responded) Kind Regards Keith Stevenson