Jump to content

Deen

Hornbill Staff
  • Posts

    782
  • Joined

  • Last visited

  • Days Won

    30

Everything posted by Deen

  1. @Adam@Greggs @Alistair Young Our developers have found the root cause of the problem and are working on a fix. I'll let you know when a patch is deployed.
  2. It appears the patch may not have completely resolved the issue. Our developers are still looking at this and another fix should be pushed out shortly.
  3. @JAquino There is possibly a defect here which will need to be corrected. I'll speak to our developers and will let you know our findings shortly.
  4. @Adrian Simpkins @HGrigsby Good to hear and apologies for the inconvenience.
  5. Hi All, Hornbill are aware of the issue and we are currently working on a solution. I'll post again shortly once I have an update on our progress.
  6. @Nikolaj @samwoo We are looking into this and are planning to patch this very shortly.
  7. @Mhari Lindsay I've just send you a PM. I'd like to see this first hand if you could please.
  8. @Mhari Lindsay I take it the affected users have tried completely refreshing their browser cache, or tried an alternative browser?
  9. Apologies all, there was an issue that has now been corrected. I'll post further details on this shortly. Deen
  10. @chriscorcoran We are aware of the issue and are working on a solution. I'll have a further update for you shortly and you can also check the status here https://status.hornbill.com/ Deen
  11. Hi All, We are aware of an issue affecting a limited number of customer instances. Our Infrastructure team are working on a solution and I will advise when a fix is in place. Deen
  12. @ssimpson There may be an issue here introduced by last nights update. We are looking into this and I will post a follow up shortly when we know more.
  13. All instances should now be back up and running. We apologise for the disruption caused here and I'll have a root cause analysis posted shortly.
  14. @jmcnae I've corrected the link. There is also https://status.hornbill.com/ that can be used to keep up to date with the outage.
  15. All, we are working on a solution and you can keep track of our progress here
  16. All, we are working on a solution and I'll have a further update as soon as possible.
  17. A limited number of instances were affected by this issue but they should all be back online. We apologise for this temporary outage and we will have a root cause available shortly.
  18. All, a patch has now been released to all instances and the issue should be resolved. Let me know if any problems remain and apologies for the inconvenience caused here.
  19. Hi all, apologies for the disruption caused here, we are looking into a solution and I will have a further update shortly.
  20. @Mark (ESC) @JanS2000 @Bev Williams A patch has now been released to all instances, you may need to flush your browser cache for the fix to take effect. Please let me know if there are any issues.
  21. A patch is currently in the works, I'll have another update shortly.
  22. @Mark (ESC) Our developers are aware of the issue and are working on a fix, I'll have a further update for you shortly.
  23. Apologies to all affected by the outage, affected instances should be coming back on line now. I will have a further update shortly with the root cause.
  24. For anyone affected, below is the root cause analysis for this disruption: At 11:18 our monitoring detected strange disk activity and high RAM usage on 1 of our NODES (mdh-p01-node16). This NODE provides core services and file attachments (Not Database records) to 5 of our customers. Investigations showed that the high RAM was a direct result of Slow disk access and all actions were therefore being effected. No obvious cause could be found, all underlying hardware was checked and no errors were detected. At 11:39 a controlled shutdown of processes was started to try and identify which process was causing the unexpected disk usage. It became clear that even with nearly all services stopped (except windows core functionality) the issue persisted. All actions taken to recover this was futile. Recovery was attempted for 30 minutes and then DR plan was initiated to restore the effected customer instances to other NODES. As the time line shows. All of these targets were achieved in less than half the RTO time. (DR Plan invoked after 30 mins, Emergency level of service within 2 hours and Restoration of Key services within 5 hours) TimeLine 11:18 Alerted by Monitoring of HIGH RAM - Investigation Began 11:19 Notified HSML of High RAM Instance and Customers still available 11:20 Investigation of Root Cause 11:30 - Xen Toolstack Restart 11:39 - Controlled Shutdown of Non Critical Processes within VM 11:45 - Controlled shutdown of ESP Services 11:50 - Instances and Customers start to become Unavailable 11:55 - All Instances on NODE unavailable. 12:01 Attempted to Restart and Correct Windows 12:02 Informed HSML of Restart of Windows 12:05 Windows Unavailable. Failed to Restart 12:15 Continued Windows Recovery - Unable to progress 12:24 Started DR Planning 12:25 Decision on New Node16 and Migrate Existing Instances to NODE17\18 14:02 All affected instances back up and running 16:52 File restore 100% DONE Total NON ER Downtime (Instance Unavailable) - From 1h7 Mins to 2H Total Recovery - From 2H to 4:52 Root Cause Due to the nature of the issue and loss of diagnostic logs that may exist in the VM we can no longer access we are unsure of the root cause. However the pattern of issues suggests some problem or corruption with the Virtual Disk containing the Windows System. Given the encrypted drives any small corruption would cause large problems. Further Planned\Required Action Rebuild NODE16 and reBalance clusters - New Node exists. Rebalance will be performed over next few weeks Investigate original failure - This will continue. We will not only be investigating the root cause (although as above with logs inside the VM being the most helpful we don't expect a hard answer), we will also be attempting to recover the NODE and its data in the hope of finding a way that should similar issue occur in future we will be able to recover quicker. Storage Servers - Already planned (Hardware already in Place and initial code changes made) was a change from having Storage (for File Attachments etc) local to the NODE. These would act in replicated pairs. Having these would mean that should we lose a NODE we don't need to restore the data from backup servers. (This would have meant once instance was created ALL was available immediately not after 2-4hours) MicroServices - Already Planned and code changes have been made over the last 2 years to support this. Along with the Storage services. The use of micro services removes the need for a home NODE box and all NODE boxes can service any instance. With this in place (And the Storage Servers) a loss of NODE is no concern (And actually part of the normal routine) We have previously never had a corruption like this and believe the chances of another occurrence are low, we also apologise for the disruption caused by the failure.
  25. For anyone affected, below is the root cause analysis for this disruption: On the morning of the 29/11 at approximately 09:15 a number of customers began reporting that their Hornbill instances were unavailable. After investigating the issue it was determined that there was a problem with the deployment of the latest Core UI framework release that caused some files from the previous version to be served along side the new release, this then lead to the old files being incorrectly cached in Cloudflare alongside files in the new release. An incompatablity between these releases lead to errors when trying to load the framework. To fix this we deployed a patch by 09:40 to break the cache again so that all requests would get the correct version. We are planning to make changes to the release process to prevent this happening again which will include purging the Cloudflare cache after deployment. We apologise for the inconvenience this will no doubt have caused.
×
×
  • Create New...