Keith Stevenson

Dear Chris, The root cause was that an automatic update of services stalled on your instance as 1 thread could not be terminated. This meant the request to STOP the main Server Service was stalled on stopping and never completed. The resolve was to kill the offending process and restart. This has highlighted an oversight in our monitoring during updates which we will now address as we should have been informed of the issue before you noticed any error so that we could take action. We are also investigating the root cause as to why the service failed to stop in a timely fashion. We hope this clarifies the issue and should you require any further information please feel free to ask. Kind Regards Keith Stevenson

Dear Chris. Thanks for the post. We can inform you that this has been resolved and you should now be able to log in. We will post an explanation shortly. Kind Regards Keith Stevenson

Kelvin, If you follow https://community.hornbill.com/forum/135-announcements/ you will receive email notifications (if configured) of all changes\updates as they become available. Kind regards Keith Stevenson

Sarah, We have now identified the root cause. Basically, 1 of your database passwords your instance uses to access your data had an illegal character in it which prevented login. This was a problem after this mornings update as the account\password is now used {each instance has several different users performing different functions and this was the first time we utilized this particular user/password} This password has now been changed. You should no longer be suffering the issue. Can you confirm. Kind Regards Keith Stevenson

Sarah, Thanks for the post. We are looking now and will give you an update within 15 minutes. Kind Regards

Cizzling, Thanks for the post. All data is stored within the geographical legal entity of your head office, for yourselves, this is currently EU with datacentres in the UK (as we are still part of EU at the moment) and all data is encrypted during transit (please see wiki.hornbill.com for further details on Data Centres). We adhere to the provisions under the Federal Privacy Act 1988 and Australian Privacy Principles along with the Australian state\territories which have their own data protection legislation in addition to the Federal and APP: Information Privacy Act 2014 (Australian Capital Territory), Information Act 2002 (Northern Territory), Privacy and Personal Information Protection Act 1998 (New South Wales), Information Privacy Act 2009 (Queensland), Personal Information Protection Act 2004 (Tasmania), and Privacy and Data Protection Act 2014 (Victoria). And this is ensured under our ISO27018 certification. These do allow transmission of data outside of Australia as long as the entity (i.e. Hornbill) has taken reasonable steps to ensure that the data is secure and again our ISO certifications are proof that we have taken extreme measures to ensure data security. As for performance. We have a development team in Australia and they report that performance is as good if not better than any other EU hosted application. There is always going to be an overhead from Australia due to their infrastructure. A ping to Australia averages around 333ms which is extremely slow and little can be done to improve that. We have however implemented local geocaching via Cloudflare for static content (i.e. Hornbill pages and applications but not your data) so that these are delivered (after the first request) from a server closer to the end user and we see that around 50% of our traffic hits these servers. The network latency issues can be seen if you login to the Hornbill.com support page and perform a status check with Show more. (Note that we have seen suboptimal performance when people use VPN to connect to the EU as this bypasses any local caching). Further information on our Data Centres, our security and controls\processes to ensure your data security are available via https://wiki.hornbill.com/index.php/FAQ:ISO Hope this answers your queries and if you have require any further information please feel free to ask. Kind Regards Keith Stevenson

This has now been resolved. We will continue to montior and provide a full review before the end of today. Kind Regards

Since 08:15 we have witnessed sub-optimal performance on around 10% of UK customers. This is due to a disk issue with the underlying hardware which we are working to resolve (ETA 20 mins). Please see status.hornbill.com for futher updates

Logan, Thanks for the post. We support the protocol (IMAP) and not a given server (ie 2016) . Therefore you should have no issues connecting to the server of your choice. Kind Regards Keith Stevenson

All, The issue should now be resolved on all instances. We will provide a full report before the end of the day. Kind Regards

All, We are 97% of effected instances fixed and we expect the remaining to be complete within next 10 minutes. We will provide a full post mortem once complete. We will update this thread in 10 minutes with an update. Kind Regards Keith Stevenson

@yelyah.nodrog Thanks for the reply. The changes are being made alphabetically and 90% of effected instances completed. Glad to hear its resolved. Kind regards

@Ann-MarieJones You are on a different pod so the changes were not applied to your instance. I will ask our Support team to contact you seperatly. Kind regards

@Martyn Houghton You instance should now be fixed. Can you confirm, kind Regards

Kevin. Thanks for the post and please see for updates

All, We have received notification of a number of customers this morning being unable to log\open or update requests within Service Manager. The root cause has been identified (database schema version infront of application version) and corrective action is currently being undertaken (Adding required column). We expect this to take around 20 minutes and will provide updates every 20 minutes in this thread. Kind regasrds Keith Stevenson

All, This was released yesterday and all instances updated overnight. The issue should no longer occur. Kind Regards Keith Stevenson

@Martyn Houghton We can inform you that this was released last night and your instance updated. Kind Regards Keith Stevenson

@Martyn Houghton Thanks for the reply. We can inform you that from dev to live usually takes around 72 hours. Kind Regards

All, Thanks for the posts. Unfortunately, this (the 400 error) is due to a unannounced change in a Google API service we utilise, the inline translation of comments returns a unexpected error (400). We have now made a change to resolve this, which is working its way through our deployment strategy (https://wiki.hornbill.com/index.php/Continuous_Delivery). Until then the translate option may not function as expected. We will inform you once the fix has been pushed to live. Kind Regards

All, Due to a unannounced change in a Google API service we utilise, the inline translation of comments returns a unexpected error (400). We have now made a change to resolve this, which is working its way through our deployment strategy (https://wiki.hornbill.com/index.php/Continuous_Delivery). Until then the translate option may not function as expected. We will inform you once the fix has been pushed to live. Kind Regards

David, Thanks for the post. Our monitoring showed no issues during the time you mentioned. Did you test the connection via the OnLine Ping Check (Need to Login to https://www.hornbill.com/support/ with your Instance ID and Email). This will allow us to narrow down where in the trip from yourselves to our servers the problem arose. Kind Regards Keith Stevenson

Aaron, Thanks for the reply. We can look into improved searching options going forward, however as all Failed will be under the Sent Items only (Thats the only folder we will place them in) then it wont need to search All. As for the large volumne and a means to reset counter I will update as soon as possible. Kind Regards Keith Stevenson

All, Thanks for the reply. We didnt expect that people would have so many failed delivery mails (certainly not 18K). We are looking into a means to clear these so you can start afresh. We will update here once we come up with a means to achieve this goal. Kind Regards Keith Stevenson

@Martyn Houghton This would apply to all historic emails. So yes you could now see that you have 30 failures but all for emails sent days or years ago. We have just made them easier to see . Kind Regards Keith Stevenson