LDAP user import error: Unable to run import, a previous import is still running

[davidrb84]

Good morning,

As previously discussed with cloud support we are getting an error (below) when trying to import a largish number of users.

I've blanked out a few bits with XXs (let me know if you need them)

For clarity we are definitely not already running an import.

I assume the next step will be to run with a force run flag to true, but wouldn't like that to be confirmed as not likely to cause more worrying issues first.

___________________________________________

 

vault-08:/var/opt/hornbill/log # cat LDAP_User_Import_2018-09-11T21-01-01+01-00.log

2018/09/11 21:01:01 [MESSAGE] ---- XMLMC LDAP Import Utility V3.0.8 ----

2018/09/11 21:01:01 [MESSAGE] Flag - config ldap-students

2018/09/11 21:01:01 [MESSAGE] Flag - logprefix

2018/09/11 21:01:01 [MESSAGE] Flag - dryrun false

2018/09/11 21:01:01 [MESSAGE] Flag - instanceid uopservicedesk

2018/09/11 21:01:01 [MESSAGE] Flag - apikey XXXXX

2018/09/11 21:01:01 [MESSAGE] Flag - apitimeout 60

2018/09/11 21:01:01 [MESSAGE] Flag - workers 1

 

2018/09/11 21:01:01 [MESSAGE] Flag - forcerun false

2018/09/11 21:01:01 [ERROR] Get https://api.github.com/repos/hornbill/goLDAPUserImport/tags: read tcp XX.XX.XX.XX:52683->192.30.253.117:443: read: connection reset by peer

2018/09/11 21:01:01 [MESSAGE] Loading Configuration Data: ldap-students

2018/09/11 21:01:01 [MESSAGE] Loading LDAP Authentication Data: 5

2018/09/11 21:01:01 [MESSAGE] Log Level 1

2018/09/11 21:01:01 [MESSAGE] Page Size 500

 

2018/09/11 21:01:01 [ERROR] Unable to run import, a provious import is still running

 

___________________________________________  

[davidrb84]

Apologies just seen the same issue in another thread, happy for these to be merged and I'll get this run with the force run flag set, if that seems the best approach?

[davidrb84]

Hi @Victor we've run a force run of a small number of users, this was successfully, however, a run of the full batch soon afterwards failed stating an import was already running.

Message from Infrastructure team "the single user import ran successfully (with the forcerun=true setting) but the next import (with forcerun=false) failed with the 'previous import is still running' message"

 

Is our next course of action to run the full (large) import with force=true.

Given this is not the behaviour described in other posts I'm cautious to do it without Hornbill agreement.

[Victor]

@davidrb84

On 9/12/2018 at 11:40 AM, davidrb84 said:

Apologies just seen the same issue in another thread, happy for these to be merged

Which thread is this? Sorry, I know is right there but can't find it ... it been a long week

2 hours ago, davidrb84 said:

Given this is not the behaviour described in other posts

Is this the same thread/post as above or different?

[davidrb84]

@Victor Sorry victor, also can't find it for the life of me. I'm confident the post indicated that if the import failed, we should set force=true. This should then complete successfully, resetting the flag to 'completed successfully'. This also mirrors what we had in the email from before.

Regardless of the original post, this appears to be the issue we've been trying to address now for 3 months.

We are now running with no new staff/students, which is proving painful.

How would be best to proceed?

[Victor]

On 9/24/2018 at 8:44 AM, davidrb84 said:

I'm confident the post indicated that if the import failed, we should set force=true.

Yes, this is correct. My advice is to run the import with the force flag enabled.

[davidrb84]

@Victor should we be concerned that we're routinely overriding what looks like a safety mechanism?

 

[davidrb84]

@Victor also we've been scheduling the student imports for nighttime as there are alot of records.

Do we need to be so cautious or will the import process either backoff if loading becomes a problem, or does it naturally work sparingly?

 

[Victor]

@davidrb84

5 minutes ago, davidrb84 said:

we're routinely overriding what looks like a safety mechanism

You should not need to do this all the time... Forcing the flag to true should be done on occasion when an import fails to complete... The next successful run should remove the failure and allow you to run the import normally the third time...

4 minutes ago, davidrb84 said:

the import process either backoff if loading becomes a problem, or does it naturally work sparingly

It should work... the issue might occur when the import is taking a bit longer (because of the amount of data) and the import is scheduled to run too often... So if there is a big amount of data then I would make sure the imports have enough time to finish between their runs... 

[davidrb84]

thanks @Victor as above, we ran the last run with 1 record, this did not reset the flag to successful (subsequent attempts errored saying it was already running) although it claimed to have been successful.

[davidrb84]

hi @Victor for the abundance of clarity, we have NOT yet carried out a mass import. When we ran an import of 1 user this did not reset the flag, as this does not match your expected behavior above I do not want to run a large import and risk causing havoc.

We will run the large import with the force=true flag if you think that appropriate.

[Victor]

@davidrb84 - it is OK to run the full import with the force flag turned ON. Please do this at your earliest convenience as I am more worried about the missing contacts/users in your system. Running the tool with the flag ON does not create any issue (in your scenario). To be of the safest side, before running the full import, check the task manager on the machine that runs the import just to ensure there is no other import process running (very unlikely there is but just to be sure).

I will continue to look into why the flag fails to reset but my main priority on this is to have the users imported.