Pre Import Actions Regex problems

[chathway]

I am trying to use the pre Import Actions within the Data import configuration section to modify several attributes being imported from AD. 

AD Attribute "StreetAddress" needs to be mapped to Hornbill site value.  Currently "streetAddress" contains carriage returns which need to be removed so lookup can match to a site location.

AD Attribute "Department" needs to be mapped to Hornbill Department and Sub-Department value.  Currently "Department" is in the format Department : Sub-department and needs to be split on the colon so lookup can match everything before colon to department and everything after to sub-department.

For "StreetAddress" I'm using the option to do a replace using \r\n in the find field and , in the replace with field however, it is not having any affect when LDAP_Import utility is run.

To get department I'm using:

(.+)(?=:)

and sub-department:

(?<=:)(.+)

 however, receive the below error:

[DEBUG] Processing LDAP User Data
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x10 pc=0x51c605]

goroutine 1 [running]:
regexp.(*Regexp).allMatches(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x64, 0xc042433138)
        /usr/local/go/src/regexp/regexp.go:683 +0xb5
regexp.(*Regexp).FindAllString(0x0, 0x0, 0x0, 0x64, 0x3f, 0xc04234e050, 0x4c)
        /usr/local/go/src/regexp/regexp.go:1024 +0x116
main.processRegexOnString(0xc042317260, 0x9, 0x0, 0x0, 0x0, 0x722848)
        /Users/TrevorK/development/GoLang/src/github.com/hornbill/goLDAPUserImport/common.go:28 +0x148
main.processImportActions(0xc04216e930, 0x0, 0xc0426cbd10)
        /Users/TrevorK/development/GoLang/src/github.com/hornbill/goLDAPUserImport/process_data.go:565 +0x457
main.processLDAPUsers()
        /Users/TrevorK/development/GoLang/src/github.com/hornbill/goLDAPUserImport/process_data.go:23 +0x1c1
main.main()
        /Users/TrevorK/development/GoLang/src/github.com/hornbill/goLDAPUserImport/main.go:83 +0x357

 Looking at the error it looks like GoLang is the flavor of Regex being used? It does appear to support ?=.  I have tried working around it but cant seem to get the desired results (tried using (.+)(?=:) for dept & (?<=:)(.+) for sub-dept)

Can anyone assist?

Many thanks

Chris

[TrevorKillick]

@chathway

Apologies the error was a crash due to a coding error for handling invalid regex, the latest version 3.1.2 has been released this morning with a fix: https://github.com/hornbill/goLDAPUserImport/releases/tag/v3.1.2

The errors you should have been getting for this are:

2018/08/13 08:07:36 [ERROR] Regex Error: error parsing regexp: invalid or unsupported Perl syntax: `(?=`
2018/08/13 08:07:36 [ERROR] Regex Error: error parsing regexp: invalid or unsupported Perl syntax: `(?<`

GoLang the language used to build the import tool does not support these regex functions. 

I managed to get both Department and Sub-Department matching with the following:

:(.*?)(?:,[A-Z]+=|$)
^[^:]+



As for the carriage return issue I tested this on Friday when Dan mentioned you were having issues replacing them have you got a text extract from AD containing an example of the text you are trying to edit?

Kind Regards

Trevor Killick

[chathway]

Thanks Trevor,

I have tried out the regex's you sent; the one for department (^[^:]+ ) works fine however, the other captures the : and space before the sub-department (and so fails lookup) .  Tried it in regex tester and can see it stripes the : off in group1 but think the import tool only sees the full match not groups?

2018/08/13 14:52:27 [DEBUG] LDAP Attribute for Site Lookup: {streetAddressClean}
2018/08/13 14:52:27 [DEBUG] Looking Up Site Concept House
Cardiff Road
2018/08/13 14:52:27 [DEBUG] Unable to Locate Site
2018/08/13 14:52:27 [DEBUG] LDAP Attribute for Org Lookup: {department}
2018/08/13 14:52:27 [DEBUG] Looking Up Org IT
2018/08/13 14:52:27 [DEBUG] Org Lookup found Id IT
2018/08/13 14:52:27 [DEBUG] LDAP Attribute for Org Lookup: {subdepartment}
2018/08/13 14:52:27 [DEBUG] Looking Up Org : IT Operations Level 2 Support
2018/08/13 14:52:27 [DEBUG] Unable to Find Organsiation : IT Operations Level 2 Support

 

Below is an extract of the streetAddress attribute from AD (attached is it pasted into notepad++ so you can see the carriage return.

Concept House
Cardiff Road

Cheers

Chris

[TrevorKillick]

@chathway

That's why in my screenshot I pass {subdepartment} into a replace action that will find and replace colon and space with an empty string. 

I will take another look and try and replicate this carriage return line feed not replacing. 

Kind Regards

Trevor Killick

[chathway]

These regex's work to do the split in the tester but not in service manager.

Department  ^[^:]+\b

Sub-department  \b[^:]+$

I have set the tester to use Golang flavor of Regex

[chathway]

11 minutes ago, TrevorKillick said:

@chathway

That's why in my screenshot I pass {subdepartment} into a replace action that will find and replace colon and space with an empty string. 

I will take another look and try and replicate this carriage return line feed not replacing. 

Kind Regards

Trevor Killick

Sorry didn't look that closely at the screenshot, I will try that now.

[chathway]

11 minutes ago, chathway said:

Sorry didn't look that closely at the screenshot, I will try that now.

@TrevorKillick that's worked now I have added the replace.

[TrevorKillick]

@chathway

That's great, with regards to the new line characters I have added a new Pre Import action of 'Trim' which will remove any pre and post whitespace as well as any new lines inside the string, this is available in the latest Import tool I just need to get the Hornbill Administration update out so you can select the action. 

I will post back once this is available.

Kind Regards

Trevor Killick

[TrevorKillick]

The Trim action is now available when configuring a Pre Import Action.

Kind Regards

Trevor Killick

[chathway]

@TrevorKillick I have updated the LDAP import tool and setup a pre import action to trim address attribute however, I just get a blank in the logs.

2018/08/17 15:17:02 [DEBUG] LDAP Attribute for Site Lookup: {streetAddressClean}
2018/08/17 15:17:02 [DEBUG] Looking Up Site [streetAddress]
2018/08/17 15:17:02 [DEBUG] Unable to Locate Site