Jump to content

Recommended Posts

Posted

@davidrb84

I can only apologise again for this situation, our platform team were investigating long into the night and have found a number of issues that they are working to fix on the Indexer Service that can cause the service to stop responding. This morning we are again pursuing this issue with the highest priority, we do understand every time this happens there is a significant impact on your instance. Both short-term workarounds we came up with yesterday ad no effect on the issue so we need to continue the investigation. 

We will provide an update to this issue as soon as we have one for you.

Kind Regards

Trevor Killick

Posted

so everyone's on the same page, we're also hoping to run a large user import very soon, but am waiting for someone to agree we should use the force=true flag before we do.

I'm obviously loath to do this import while we're also seeing these issues of unknown cause.

Any comment on if we should do that import or not while investigating this would be welcomed

@Victor

 

Posted

@davidrb84

I just wanted to give you an update.  Firstly I wanted to assure you that we have now escalated this as our number one development priority, we have a cross-functional team looking at the problem from a number of angles and we are of course committed to getting to the bottom of it as quickly as possible.  Its a technically complicated issue that almost certainly exists within a third-party component, and while we have the source code, getting to grips with the design intent is one of the problems we face.  We have a number of threads of activity ongoing. First we are looking to identify and isolate the specific problem to alleviate the short-term problem, second, we are looking to identify a code fix, possibly in the third party component, and lastly, I am looking at alternative strategies/approaches to our architecture to achieve the same or improved implementation.  In all cases I just wanted you to know that this is as important to us as it is to you and we will not stop until we have a resolution.  Thank you for your ongoing feedback and your patience. 

Gerry

Posted

@davidrb84 What time have you scheduled your imports to run? From a technical point of view, the user imports do not involve any indexing so it should not cause performance issues for you at the moment but it's always best to run the import once things have settled. Could this wait until we get to the bottom of this?

Thanks

Pamela

Posted

@Pamela Hi Pamela, the imports are a bit of a sore point as it's not run for 8+ weeks meaning we've got 10k or so customers missing (out of 35k or so ideal), while causing us major issues another day or so shouldn't make my team hate me too much more.

Posted

@davidrb84 This evening our Development team will carry out some maintenance work on your instance. We are hoping for some feedback tomorrow regarding how the system will be. We will also increase the level of logging so that if it continues happening tomorrow we will have some more helpful information to help resolve this.

Thanks

Pamela

Posted

@davidrb84

We ended up rebuilding your index which solved the immediate problem.  We do have a software problem that we can sporadically re-create, this can cause our index writer process to fail which seemingly *can* corrupt the index when the system is under heavy load.  By doing a full rebuild of the index we solved your immediate problem.  We are looking at ways of fixing the other issues which generally do not show up in production, so that's probably a longer-term prospect.  For now, I would suggest you should consider the problem is resolved unless it reoccurs in the next couple of days. 

My apologies for the disruption this problem caused you,  if its any consolation to know, we have not seen this problem before so it is as new to us as it was to you. I would not expect this to be a problem in the future, and on the odd chance it does re-occur on your instance (or any other instance for that matter) its now a documented known issue and we would be able to respond with a resolution much faster than we were able to this time. Hope that's ok. 

Gerry

Posted

@Gerry

Thank-you Gerry I appreciate you taking a personal interest in this issue, and it is nice to be novel :-)

Please pass on my appreciation to the team, sounds like they've gone above and beyond on this one!

Posted

@davidrb84

You are most welcome. I take an interest in every single issue that is escalated because in truth we actually have very few of them.  The effort in design and implementation of our platform really paid off in that regard.  I and the team will always go above and beyond when there is a serious problem.  We are not perfect and we can get things wrong ourselves from time to time,  but it is our obligation to provide an outstanding service so when things don't go well it matters a lot to us. I am extremely grateful to you and our other customers who always give us the space to resolve issues when they arise, that is worth its weight in gold and I cannot tell you how much we appreciate that when we are under pressure this end.  That sense of mutual partnership and trust makes all the difference.

Thanks,

Gerry

  • Sad 1
Posted

@Jeremy With regards to the 'could not initialise core data - error code' you raised last Thursday, have you seen this error again in the past few days?

Thanks

Pamela

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...