TrevorKillick Posted October 2, 2018 Share Posted October 2, 2018 @davidrb84 I can only apologise again for this situation, our platform team were investigating long into the night and have found a number of issues that they are working to fix on the Indexer Service that can cause the service to stop responding. This morning we are again pursuing this issue with the highest priority, we do understand every time this happens there is a significant impact on your instance. Both short-term workarounds we came up with yesterday ad no effect on the issue so we need to continue the investigation. We will provide an update to this issue as soon as we have one for you. Kind Regards Trevor Killick Link to comment Share on other sites More sharing options...
davidrb84 Posted October 2, 2018 Author Share Posted October 2, 2018 Thanks @Pamela @TrevorKillick we do appreciate the team's hard work. 1 Link to comment Share on other sites More sharing options...
davidrb84 Posted October 2, 2018 Author Share Posted October 2, 2018 so everyone's on the same page, we're also hoping to run a large user import very soon, but am waiting for someone to agree we should use the force=true flag before we do. I'm obviously loath to do this import while we're also seeing these issues of unknown cause. Any comment on if we should do that import or not while investigating this would be welcomed @Victor Link to comment Share on other sites More sharing options...
Jeremy Posted October 2, 2018 Share Posted October 2, 2018 @Pamela we have just got this error in our instance when trying to mark an email as read, and it doesn't allow you. Link to comment Share on other sites More sharing options...
Jeremy Posted October 2, 2018 Share Posted October 2, 2018 Just trying to add an update to a request Link to comment Share on other sites More sharing options...
Jeremy Posted October 2, 2018 Share Posted October 2, 2018 @Pamela trying to log an email request Link to comment Share on other sites More sharing options...
Jeremy Posted October 2, 2018 Share Posted October 2, 2018 Link to comment Share on other sites More sharing options...
Jeremy Posted October 2, 2018 Share Posted October 2, 2018 Also cannot find requests when applying email updates @Pamela even if we search for them Link to comment Share on other sites More sharing options...
Gerry Posted October 2, 2018 Share Posted October 2, 2018 @davidrb84 I just wanted to give you an update. Firstly I wanted to assure you that we have now escalated this as our number one development priority, we have a cross-functional team looking at the problem from a number of angles and we are of course committed to getting to the bottom of it as quickly as possible. Its a technically complicated issue that almost certainly exists within a third-party component, and while we have the source code, getting to grips with the design intent is one of the problems we face. We have a number of threads of activity ongoing. First we are looking to identify and isolate the specific problem to alleviate the short-term problem, second, we are looking to identify a code fix, possibly in the third party component, and lastly, I am looking at alternative strategies/approaches to our architecture to achieve the same or improved implementation. In all cases I just wanted you to know that this is as important to us as it is to you and we will not stop until we have a resolution. Thank you for your ongoing feedback and your patience. Gerry Link to comment Share on other sites More sharing options...
Guest Posted October 2, 2018 Share Posted October 2, 2018 @davidrb84 What time have you scheduled your imports to run? From a technical point of view, the user imports do not involve any indexing so it should not cause performance issues for you at the moment but it's always best to run the import once things have settled. Could this wait until we get to the bottom of this? Thanks Pamela Link to comment Share on other sites More sharing options...
davidrb84 Posted October 2, 2018 Author Share Posted October 2, 2018 @Pamela Hi Pamela, the imports are a bit of a sore point as it's not run for 8+ weeks meaning we've got 10k or so customers missing (out of 35k or so ideal), while causing us major issues another day or so shouldn't make my team hate me too much more. Link to comment Share on other sites More sharing options...
Guest Posted October 2, 2018 Share Posted October 2, 2018 @davidrb84 This evening our Development team will carry out some maintenance work on your instance. We are hoping for some feedback tomorrow regarding how the system will be. We will also increase the level of logging so that if it continues happening tomorrow we will have some more helpful information to help resolve this. Thanks Pamela Link to comment Share on other sites More sharing options...
davidrb84 Posted October 2, 2018 Author Share Posted October 2, 2018 @Pamela thanks Pamela, it is appreciated, let us know if there's anything you'd like us to do. Link to comment Share on other sites More sharing options...
davidrb84 Posted October 3, 2018 Author Share Posted October 3, 2018 Hi @Pamela hope you're well. We've had no issues today, do we know if last night's efforts were successful? Link to comment Share on other sites More sharing options...
Gerry Posted October 3, 2018 Share Posted October 3, 2018 @davidrb84 We ended up rebuilding your index which solved the immediate problem. We do have a software problem that we can sporadically re-create, this can cause our index writer process to fail which seemingly *can* corrupt the index when the system is under heavy load. By doing a full rebuild of the index we solved your immediate problem. We are looking at ways of fixing the other issues which generally do not show up in production, so that's probably a longer-term prospect. For now, I would suggest you should consider the problem is resolved unless it reoccurs in the next couple of days. My apologies for the disruption this problem caused you, if its any consolation to know, we have not seen this problem before so it is as new to us as it was to you. I would not expect this to be a problem in the future, and on the odd chance it does re-occur on your instance (or any other instance for that matter) its now a documented known issue and we would be able to respond with a resolution much faster than we were able to this time. Hope that's ok. Gerry Link to comment Share on other sites More sharing options...
davidrb84 Posted October 3, 2018 Author Share Posted October 3, 2018 @Gerry Thank-you Gerry I appreciate you taking a personal interest in this issue, and it is nice to be novel :-) Please pass on my appreciation to the team, sounds like they've gone above and beyond on this one! Link to comment Share on other sites More sharing options...
Gerry Posted October 3, 2018 Share Posted October 3, 2018 @davidrb84 You are most welcome. I take an interest in every single issue that is escalated because in truth we actually have very few of them. The effort in design and implementation of our platform really paid off in that regard. I and the team will always go above and beyond when there is a serious problem. We are not perfect and we can get things wrong ourselves from time to time, but it is our obligation to provide an outstanding service so when things don't go well it matters a lot to us. I am extremely grateful to you and our other customers who always give us the space to resolve issues when they arise, that is worth its weight in gold and I cannot tell you how much we appreciate that when we are under pressure this end. That sense of mutual partnership and trust makes all the difference. Thanks, Gerry 1 Link to comment Share on other sites More sharing options...
Jeremy Posted October 11, 2018 Share Posted October 11, 2018 @Pamela We are experiencing these issues again please can this be investigated. Link to comment Share on other sites More sharing options...
Jeremy Posted October 11, 2018 Share Posted October 11, 2018 In the logs this is constant, not sure if it's relevant Link to comment Share on other sites More sharing options...
Jeremy Posted October 11, 2018 Share Posted October 11, 2018 Link to comment Share on other sites More sharing options...
Victor Posted October 11, 2018 Share Posted October 11, 2018 @Jeremy - we are aware of the issue, one of our data services is under heavy load. We're working on this, will update ASAP. Link to comment Share on other sites More sharing options...
Guest Posted October 15, 2018 Share Posted October 15, 2018 @Jeremy With regards to the 'could not initialise core data - error code' you raised last Thursday, have you seen this error again in the past few days? Thanks Pamela Link to comment Share on other sites More sharing options...
Jeremy Posted October 15, 2018 Share Posted October 15, 2018 @Pamela No we have not seen this over the last few days Link to comment Share on other sites More sharing options...
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now