Keith Stevenson

Steph, Thanks for the post. All our monitoring (from 6 different locations worldwide) saw no issue and the instance appears to have been fine.. Also the status.hornbill.com was showing an partial outage as it hadnt been updated since an issue 3 weeks ago and unrealated. We have now set status to be correct. Kind Regards Keith Stevenson

Rob, For the first part, Only the Cloud team (currently 4 members ) can even access systems that may contain customer data (Unless you grant our Support Team access via Key which you control what they can perform and how long) . No other staff have access (All Customer data is in DCs that all access is prevented except for the Cloud Team). Any access to machines that may contain data is further controlled and any login will automatically post to a Hornbill Workspace for all the Company to see. This Login and associated Incidents are then audited and has to be associated with a valid reason (Incident\Change Request etc or disciplinary action can occur. ). Our training\Processes and controls ensure that we only access machines hosting data when absolutely necessary. All this is covered under our ISO controls which are audited annually by the BSI . (See wiki.hornbill.com for a overview of our ISO policies, if your Enterprise customer you can also get a yearly 1 to 1 review of our controls and audits) So if a member of the Cloud Team (or if their accounts were compromised) had access to a machine hosting data it would be flagged instantly. Even then they would not be able to decode the KeySafe Credentials held in Hornbill which would be needed to invoke the ITOM integration As for the 2nd part of your query. The ITOM job will do whatever you configure it to do, so providing you pass the correct ID etc it will do the expected action on the given account. Hope this clarifies your query Kind Regards Keith Stevenson Hornbill Cloud Team

@chriscorcoran From the logs we can see that email is being sent currently but there was a period early when your external SMTP Service was down. You should contact the owner of the configured SMTP service for further details. Kind Regards Keith Stevenson

All, Thanks for the posts. As above we can see no delay or unexpected load on your instances. Can you run Chrome web browser in Developer Tools (F12) and replicate the issue then , right click as save ALL As HAR File (Then send this via DM to myself). Kind Regards

@LifeOfJonny Can you try that now Kind Regard

Good to hear. Everyone should be back and running. KInd Regards

@Berto2002 Can you click the Disable Cache option and try again Kind Regards

We have pushed a change. Can anyone confirm the issue is now working as expected. Kind Regards

@Martyn Houghton Thanks for the reply. Can you try Dev F12 console and see if there are any errors. Also In F12 console on the network tab click Dsiable Cache and see if that changes it. (Our Developers are still working on root cause) Kind Regards

All Thanks for the post. Can you clear your Browsers Cache. This should allow it to conitnue.. We are looking at the root cause Kind Regards Hornbill Cloud team

Breto, Thanks for the post. Can you clear your Browsers Cache. Kind Regards Hornbill Cloud team

Oscar Thansk for the reply. We have had no issues with direct email to Office365 hosted domains, the way that works is different. And the openSSL client is a open source tool not part of Hornbill (so outside). As for other customers I believe around 4% have reported issues, but given the intermittent nature of the issue others may not have seen a problem as they are eventually sent. 1 thing about your logs is that on the occasions when MS does respond (its about 50% of requests ) they show a OAuth failure so you may have a secondary issue and you should check that (Although OAuth is not required for SMTP and MS are not removing it so we would recommend keeping Auth for Outbound as Classic ) Having just tried for 30 minutes attempting to connect and send different tools its about 1/4 success .. As Hornbill does the same thing every time we send an email (and in this case I was using the test connection option) this is a MS issue that is intermittent (most likely because they load balance the SMTP servers and only in 1/4 times do you get email sent. If this was Hornbill we would expect 100% failure Kind Regards

Oscar, Thanks for the post. Sadly as this is a MS issue we are not able to comment on their reporting or issue. All we can say is that when we make a connection the TLS handshake fails. The below when you connect should allow you to say EHLO xxxxxx (Where XXXX is your servername in our case live.hornbill.com) . This hangs (and not just from our servers but from anywhere we have tried this) openssl s_client -connect smtp.office365.com:587 -starttls smtp If you compare this to say googles GMAIL you can see that this works as expected and on entering EHLO it shows list of options and allows you to carry on with the connection openssl s_client -connect smtp.gmail.com:587 -starttls smtp We are not a MS customer and therefore cannot raise the issue with them. As for alternatives, Direct Outbound rather than SMTP SMART host (See https://wiki.hornbill.com/index.php?title=Outbound_Mail_Routing) will prevent this as we go direct to the given mail server rather than via a 3rd Party who we have no control over Kind Regards Hornbill Cloud Team

Euan, Thanks for the post. We can see no performance issues for your instance and no slow API (All your API requests are < 1 second) .. Can you complete the Instance Health Check from within the Hornbill Support portal as that will show further details and provide details from the Network TAB of a browser showing slow API Kind Regards Hornbill

@Berto2002 We can not see any errors in the Server at the times above.. However we would not expect to see any as -1 means that the API request probably didnt make it to the server . The API that failed to reach our server could be any number and we would need to see the full Network tab from F12 developer tools to see which specific API, the payload and the target. Kind Regards Hornbill Cloud Team

@Martyn Houghton The Updates for Platform will be at 5am UTC and Apps at 0530 UTC all year round. Apps updates should not cause downtime. This ongoing change is all a journey to micro services in which there is no concept of nodes as exist now and once achieved their will be no update time at all (As micro services will just start when required with the latest versions and those micro services that were running previous version will disappear once they have completed the current task) This update time applies to all DCs no matter where the location Kind Regards Keith Stevenson

@Martyn Houghton The platform setting was removed from being an instance setting more than 16 months ago and since then all platform updates have been at 5AM (These are the ones that may cause upto 1 minute of interruption. ) Applications up until last month were also at the same time (Due to the configurable setting being ignored) and these should not cause any interruption. Going forward these will be at 0530 (Above I originally stated 0600 but will correct that) and typically take around 30 seconds (But with 0 interruption) . I can agree that there was no notification of the changes and perhaps it would have been better to post before the change, but as the Platform setting was only ever changed by 1 instance and the application setting never took effect as expected (until last month) it was deemed unnecessary. In hindsight it would have probably been better to remove the option for application setting last month rather than correct the existing issue (which may have given a false impression) and going forward we will provide more detailed announcements when not covered by release notes. Kind Regards Keith Stevenson

@Martyn Houghton All Platform Updates are performed at 5am UTC Mon-Fri. This is not customisable. Going forward (by the end of this month) the option for Application updates will also be removed and these too will be performed at 6am UTC. The reasons are many but any interruption during the update should be limited to no more than 1 minute. Kind Regards Keith Stevenson

@Martyn Houghton All Platform Updates are performed at 5am UTC Mon-Fri. This is not customisable. Going forward (by the end of this month) the option for Application updates will also be removed and these too will be performed at 6am UTC. The reasons are many but any interruption during the update should be limited to no more than 1 minute. Kind Regards Keith Stevenson

Ben, Thanks for the reply. Outbound email will not be effected by the MS Change. Kind Regards Keith Stevenson

@Ben Maddams Please see below (Note that we did email all Primary Technical and Secondary on September 23rd about this and you should discuss with them the below notification) Kind Regards

@Nikolaj Good to hear. If you have any more examples (Please provide date\time of Email and expected Rule that would match) we can take a look but everything does appear to be working as intended. Kind Regards

@Nikolaj I will chase Dev today. Kind Regards

@Nikolaj From the logs it appears that the one this monring (Subject like MailSweeper block Info) failed as 1 or more accounts have the same email address and therefore it cant find which customer to use.. (This was sent from your email address) . The previous email recieved (at 03:54 subject 31053 srv-dagt-o1 Down) was processed as expected The fix for the first one is to ensure only 1 customer has the email address, Kind Regards

All, The current permissions required for Microsoft OAuth are: Mail.Read - Read User Mail Mail.Read.All - Read User and Shared Mail Mail.Read.Shared - Read User and Shared Mail Mail.Send - Send Mail as a User Mail.Send.All - Send Mail on behalf of others Mail.Send.Shared - Send Mail on behalf of others USer.Read - Read Users Profiles We are creating a new OAuth type which will remove the SEND requirement from the above for those customers who have stricter polices. KInd Regards Hornbill Cloud Team