Deen
-
Posts
783 -
Joined
-
Last visited
-
Days Won
30
Content Type
Profiles
Forums
Enhancement Requests
Posts posted by Deen
-
-
@Adam@Greggs @Alistair Young Our developers have found the root cause of the problem and are working on a fix. I'll let you know when a patch is deployed.
-
3
-
-
It appears the patch may not have completely resolved the issue. Our developers are still looking at this and another fix should be pushed out shortly.
-
2
-
-
@JAquino There is possibly a defect here which will need to be corrected. I'll speak to our developers and will let you know our findings shortly.
-
1
-
-
@Adrian Simpkins @HGrigsby Good to hear and apologies for the inconvenience.
-
1
-
-
Hi All,
Hornbill are aware of the issue and we are currently working on a solution. I'll post again shortly once I have an update on our progress.
-
2
-
-
-
@Mhari Lindsay I've just send you a PM. I'd like to see this first hand if you could please.
-
@Mhari Lindsay I take it the affected users have tried completely refreshing their browser cache, or tried an alternative browser?
-
Apologies all, there was an issue that has now been corrected. I'll post further details on this shortly.
Deen
-
1
-
3
-
-
@chriscorcoran We are aware of the issue and are working on a solution. I'll have a further update for you shortly and you can also check the status here https://status.hornbill.com/
Deen
-
Hi All,
We are aware of an issue affecting a limited number of customer instances. Our Infrastructure team are working on a solution and I will advise when a fix is in place.
Deen-
2
-
-
@ssimpson There may be an issue here introduced by last nights update. We are looking into this and I will post a follow up shortly when we know more.
-
All instances should now be back up and running. We apologise for the disruption caused here and I'll have a root cause analysis posted shortly.
-
@jmcnae I've corrected the link. There is also https://status.hornbill.com/ that can be used to keep up to date with the outage.
-
All, we are working on a solution and you can keep track of our progress here
-
All, we are working on a solution and I'll have a further update as soon as possible.
-
A limited number of instances were affected by this issue but they should all be back online. We apologise for this temporary outage and we will have a root cause available shortly.
-
1
-
-
All, a patch has now been released to all instances and the issue should be resolved. Let me know if any problems remain and apologies for the inconvenience caused here.
-
2
-
-
Hi all, apologies for the disruption caused here, we are looking into a solution and I will have a further update shortly.
-
7
-
-
@Mark (ESC) @JanS2000 @Bev Williams A patch has now been released to all instances, you may need to flush your browser cache for the fix to take effect. Please let me know if there are any issues.
-
2
-
-
A patch is currently in the works, I'll have another update shortly.
-
2
-
-
@Mark (ESC) Our developers are aware of the issue and are working on a fix, I'll have a further update for you shortly.
-
4
-
2
-
-
Apologies to all affected by the outage, affected instances should be coming back on line now. I will have a further update shortly with the root cause.
-
4
-
-
For anyone affected, below is the root cause analysis for this disruption:
At 11:18 our monitoring detected strange disk activity and high RAM usage on 1 of our NODES (mdh-p01-node16).
This NODE provides core services and file attachments (Not Database records) to 5 of our customers. Investigations showed that the high RAM was a direct result of Slow disk access and all actions were therefore being effected.
No obvious cause could be found, all underlying hardware was checked and no errors were detected. At 11:39 a controlled shutdown of processes was started to try and identify which process was causing the unexpected disk usage. It became clear that even with nearly all services stopped (except windows core functionality) the issue persisted. All actions taken to recover this was futile.
Recovery was attempted for 30 minutes and then DR plan was initiated to restore the effected customer instances to other NODES. As the time line shows. All of these targets were achieved in less than half the RTO time. (DR Plan invoked after 30 mins, Emergency level of service within 2 hours and Restoration of Key services within 5 hours)
TimeLine
11:18 Alerted by Monitoring of HIGH RAM - Investigation Began
11:19 Notified HSML of High RAM
Instance and Customers still available
11:20 Investigation of Root Cause
11:30 - Xen Toolstack Restart
11:39 - Controlled Shutdown of Non Critical Processes within VM
11:45 - Controlled shutdown of ESP Services
11:50 - Instances and Customers start to become Unavailable
11:55 - All Instances on NODE unavailable.
12:01 Attempted to Restart and Correct Windows
12:02 Informed HSML of Restart of Windows
12:05 Windows Unavailable. Failed to Restart
12:15 Continued Windows Recovery - Unable to progress
12:24 Started DR Planning
12:25 Decision on New Node16 and Migrate Existing Instances to NODE17\18
14:02 All affected instances back up and running
16:52 File restore 100% DONE
Total NON ER Downtime (Instance Unavailable) - From 1h7 Mins to 2H
Total Recovery - From 2H to 4:52
Root Cause
Due to the nature of the issue and loss of diagnostic logs that may exist in the VM we can no longer access we are unsure of the root cause. However the pattern of issues suggests some problem or corruption with the Virtual Disk containing the Windows System. Given the encrypted drives any small corruption would cause large problems.
Further Planned\Required Action
Rebuild NODE16 and reBalance clusters - New Node exists. Rebalance will be performed over next few weeks
Investigate original failure - This will continue. We will not only be investigating the root cause (although as above with logs inside the VM being the most helpful we don't expect a hard answer), we will also be attempting to recover the NODE and its data in the hope of finding a way that should similar issue occur in future we will be able to recover quicker.
Storage Servers - Already planned (Hardware already in Place and initial code changes made) was a change from having Storage (for File Attachments etc) local to the NODE. These would act in replicated pairs. Having these would mean that should we lose a NODE we don't need to restore the data from backup servers. (This would have meant once instance was created ALL was available immediately not after 2-4hours)
MicroServices - Already Planned and code changes have been made over the last 2 years to support this. Along with the Storage services. The use of micro services removes the need for a home NODE box and all NODE boxes can service any instance. With this in place (And the Storage Servers) a loss of NODE is no concern (And actually part of the normal routine)
We have previously never had a corruption like this and believe the chances of another occurrence are low, we also apologise for the disruption caused by the failure.
Amendments to Layout of Priority / Urgency in Incidents
in Service Manager
Posted
@JAquino Please note that there is a new Service Manager release that should be live Tuesday morning. This release contains a fix for the Priority status issues.