[Hornbill service disruption]

For anyone affected, below is the root cause analysis for this disruption:

On the morning of the 29/11 at approximately 09:15 a number of customers began reporting that their Hornbill instances were unavailable.  

After investigating the issue it was determined that there was a problem with the deployment of the latest Core UI framework release that caused some files from the previous version to be served along side the new release, this then lead to the old files being incorrectly cached in Cloudflare alongside files in the new release.  An incompatablity between these releases lead to errors when trying to load the framework.
  
To fix this we deployed a patch by 09:40 to break the cache again so that all requests would get the correct version. We are planning to make changes to the release process to prevent this happening again which will include purging the Cloudflare cache after deployment.

We apologise for the inconvenience this will no doubt have caused.

[Service Manager is offline]

The recovery process is now complete and all instances have been fully restored.  We will be in touch shortly with an RCA and apologise for the disruption.

[Service Manager is offline]

For any instances still affected by missing files, the recovery process is still ongoing and we expect this to be complete in the next 30 minutes or so.  I will confirm here when this has been completed.

[Outbound email failures]

After additional tests this appears to be an issue with Office365 where intermittently it is unable to complete a valid connection.  Any emails stuck in your Outbox will have 9 retry attempts, if any of these are successful they will leave the mailbox as expected - you will see from your Sent Items folder that emails are sending, it is a few specific ones that fail.  You can also manually resend affected mails using the envelope icon if they are time sensitive.

We have had reports from a number of other Customers with the same issue, so there does seem to be a problem with Office365 mail at the moment.

[Outbound email failures]

A number of users are reporting intermittent issues with outbound email stuck in the Outbox.  We are currently looking into the root cause and I will have a further update shortly.

[LDAP user import Question.]

@Paul Morrow If the LDAP import was working fine yesterday (and I presume it has been fine for some time) has their been any changes to the import in the last day or so that might have caused this?

[Report on Accounts]

@BobbyB If you can show me how the report has been set up I'll see what I can suggest.

[When using Decision Node the Variable Picker does not show all prior objects]

@Martyn Houghton This may possibly be a defect, it would be worth raising a support request to confirm this one way or the other.

[Hornbill down again....]

All, we will be performing an update at 11.30am which should correct all the issues that have been reported this morning.  However this will result in a temporary outage of a couple of minutes.  We apologise for the inconvenience this will cause.

[Unable to logon to system and unable to raise a support ticket]

All, we are aware that some may be having further issues here this morning.  We are looking at this with the highest priority.

[Application Rights Tool Tips - make the useful please]

@Martyn Houghton I've asked our developers to look into adding these, and also to add additional content rather than mirroring the name of the right in question.

[Application Rights Documentation]

@Martyn Houghton I'll look to make the requested additions to the wiki over the coming days for Application Rights.

[issue after updates last night?]

@Caroline @Martyn Houghton @John C @Rashid.Ahmed We have now corrected this issue and are in the process of deploying a patch which will be done shortly.  Apologies for any inconvenience caused by this.

[Tickets showing as assigned to Owner when they are not]

@Sam P Have you completely flushed your browser cache? you should not be seeing the problem after that.

[Tickets showing as assigned to Owner when they are not]

All, we have pushed out a fix for this, it would be worth flushing your browser cache then checking to see if this remains a problem.

[Employee Portal header background image reverted after update]

Hi Tracey, thanks for the update.  Good to hear your back up and running.

Apologies for the disruption yesterday.

[Employee Portal header background image reverted after update]

We currently have a patch available for this issue.  Some downtime, possibly 10 minutes or so will be required to apply the patch and as a result it would probably be preferable to apply this later this tonight.  If anyone feels however that this cannot wait please let me know.

[Reports failing with a "The maximum columns of 20 for this report has been exceeded. Requested number is 116"]

A patch was applied to all instances yesterday that should have resolved this issue for all affected instances.  Please let me know however if any issues persist.

[Reports failing with a "The maximum columns of 20 for this report has been exceeded. Requested number is 116"]

As a further update to this, we have temporarily upped the column limit to 200 and there will also be a fix in the next ESP release as the column restriction was not meant to apply to all report types.

[Reports failing with a "The maximum columns of 20 for this report has been exceeded. Requested number is 116"]

@RobW @Dan Munns @HGrigsby This is a change that is covered in the latest ESP release notes:
 

• Stops the system from processing the reports further when the maximum is reached and to return an error report as the output. Updated the HTML templates for the reports so that the error message can be produced and displayed.

Now I believe that previously these reports would fail silently, if you correct the column count to match the configured limit there shouldn't be a problem.

[Cannot download the Hornbill Meta data for user, admin or service....]

@John C I believe she did.  Although I am not yet sure of the root cause.

[Email Sent Item Purging in ESP build 3511 - What is the setting name and when will it trigger?]

This was added in error to the release notes and the notes have been corrected this morning to reflect this, as the feature is not actually present in the current release.

I suspect it will be in an upcoming release although I don't yet have the details of when that might be.

[Issue with Reports]

@Darren Rose I take it we are talking about widgets here rather than actual reports?  We have looked at some of the heavier ones and the largest contains the following in the query:

h_itsm_requests.h_datelogged >= '2019-04-30 00:00:00'

Is there a need for the query to bring back requests from early 2019?  If this can be reduced to a weekly or monthly value that should significantly reduce the query time.  Also looking at the query two of the fields h_fk_servicename and h_catalog don't have indexes, you could swap them with h_fk_serviceid and h_catalog_id as that should also help to speed up the query and get it under the current threshold.

[Requests raised around 3am are starting to fail again]

@Adrian Simpkins If you could provide the details of the two impacted requests we can look into this further.  It might be best if you do this via a new request though.

[Unable to delete an image in update]

@Caroline I believe if the following Service Manager setting is set as required then the user that created the update should be able to edit it, as long as the conditions in the description below are met: