Missing users with Azure Import

[Chris]

Hi - I have a problem I'm struggling to solve whereby the Azure Import script isn't importing all the members of an AAD group.

• I've set up the Azure User Import as per the Wiki instructions (https://wiki.hornbill.com/index.php/Azure_User_Import) with this working successfully when the import script (conf.json) reads an AAD group with 20 members. 
• I've then added another 100 users to the group and re-run the script (many times!) but not all users are imported/created in our Hornbill instance (about 40 were missing). 
• As part of my trouble shooting I created another AAD group, added the missing users and amended the script to reference this new group instead - on running it these users get imported without any issues. 
• I reverted the script to the intended AAD group, added 5 extra users to the group and re-ran, but only 1 of the new 5 users get imported/created.

Anyone able to offer any insight or solutions?  Any help really appreciated!

Chris

[SamS]

Hi @Chris,

Does the log file provide any hint about what went wrong?

[Chris]

Hi @SamS - thanks for your quick reply.  I can't find any clues from the log file.  I'm pasting exerts (with a couple of annotations) below.  Entries have been redacted in places for obvious reasons.

2020/05/13 12:48:28 [DEBUG] ---- XMLMC SQL Import Utility V2.4.1 ----
2020/05/13 12:48:28 [DEBUG] Flag - Config File conf.json
2020/05/13 12:48:28 [DEBUG] Flag - Zone eur
2020/05/13 12:48:28 [DEBUG] Flag - Log Prefix 
2020/05/13 12:48:28 [DEBUG] Flag - Dry Run false
2020/05/13 12:48:28 [DEBUG] Flag - Workers 1
2020/05/13 12:48:28 [MESSAGE] ---- XMLMC DB User Import Utility V2.4.1 ----
2020/05/13 12:48:28 [MESSAGE] Flag - config goAzure2HUserImport
2020/05/13 12:48:28 [MESSAGE] Flag - logprefix 
2020/05/13 12:48:28 [MESSAGE] Flag - dryrun false
2020/05/13 12:48:28 [MESSAGE] Flag - instanceid 
2020/05/13 12:48:28 [MESSAGE] Flag - apikey 
2020/05/13 12:48:28 [MESSAGE] Flag - apitimeout 60
2020/05/13 12:48:28 [MESSAGE] Flag - workers 1

2020/05/13 12:48:28 [MESSAGE] Flag - forcerun false
2020/05/13 12:48:32 [DEBUG] Loading Config File: C:\Hornbill_Import/conf.json
2020/05/13 12:48:33 [DEBUG] Processing Old Log Files Current Retention Set to: 0
2020/05/13 12:48:33 [MESSAGE] Querying Group [*********************]
2020/05/13 12:48:33 [MESSAGE] [Azure] Query Azure Data using Graph API. Please wait...
2020/05/13 12:48:33 [DEBUG] [SCRIPT] Generating Bearer Token
2020/05/13 12:48:33 [DEBUG] [SCRIPT] Got New BearerToken
2020/05/13 12:48:33 [DEBUG] [AZURE] API URL: https://graph.microsoft.com/v1.0/groups/*******************************/members?
2020/05/13 12:48:34 [MESSAGE] [Azure] Connection Successful
2020/05/13 12:48:34 [DEBUG] [SCRIPT] Re-using BearerToken
2020/05/13 12:48:34 [DEBUG] [AZURE] User API URL: https://graph.microsoft.com/v1.0/users/*************************
2020/05/13 12:48:34 [MESSAGE] [Azure] Connection Successful
2020/05/13 12:48:34 [DEBUG] [Azure] Querying Azure Manager. Please wait...

{EDIT} - removed the hundred+ users listed in the log, but the example below is the user that is in the ADD group but does not get imported/created in the Hornbill instance and isn't mentioned again elsewhere in the log

2020/05/13 12:49:18 [DEBUG] [Azure] Querying Azure Manager. Please wait...
2020/05/13 12:49:18 [DEBUG] [SCRIPT] Re-using BearerToken
2020/05/13 12:49:18 [DEBUG] [AZURE] API URL: https://graph.microsoft.com/v1.0/users/Firstname.Lastname@oursmtpdomain.uk/manager
2020/05/13 12:49:19 [DEBUG]  [Azure] Response: No manager found for Firstname.Lastname@oursmtpdomain.uk
2020/05/13 12:49:19 [DEBUG] [SCRIPT] Re-using BearerToken
2020/05/13 12:49:19 [DEBUG] [AZURE] User API URL: https://graph.microsoft.com/v1.0/users/*************************
2020/05/13 12:49:19 [MESSAGE] [Azure] Connection Successful

{EDIT}

2020/05/13 12:49:25 [DEBUG]  [Azure] Determined Token: X%274453707409000100000000000********************************************
2020/05/13 12:49:25 [MESSAGE] [Azure] Found 100 Users
2020/05/13 12:49:25 [MESSAGE] [Azure] Query Azure Data using Graph API. Please wait...
2020/05/13 12:49:25 [DEBUG] [SCRIPT] Re-using BearerToken
2020/05/13 12:49:25 [DEBUG] [AZURE] API URL: https://graph.microsoft.com/v1.0/groups/37ab571d-993b-424c-9e84-f2ee4c3e3893/members?&$skiptoken=X'44537074090001000000000************************************
2020/05/13 12:49:26 [MESSAGE] [Azure] Connection Successful

{EDIT}

2020/05/13 12:49:42 [DEBUG] Loading Users from Hornbill
2020/05/13 12:49:42 [DEBUG] getUserAccountsList Count: 144
2020/05/13 12:49:42 [DEBUG] Loading User Accounts List Offset: 0

2020/05/13 12:49:42 [DEBUG] Loading User Accounts List Offset: 100

2020/05/13 12:49:42 [DEBUG] Users Loaded: 144
2020/05/13 12:49:42 [DEBUG] Loading Users Roles from Hornbill
2020/05/13 12:49:42 [DEBUG] getUserAccountsRolesList Count: 505
2020/05/13 12:49:42 [DEBUG] Loading User Accounts Roles List Offset: 0
2020/05/13 12:49:42 [DEBUG] Loading User Accounts Roles List Offset: 100
2020/05/13 12:49:42 [DEBUG] Loading User Accounts Roles List Offset: 200
2020/05/13 12:49:42 [DEBUG] Loading User Accounts Roles List Offset: 300
2020/05/13 12:49:42 [DEBUG] Loading User Accounts Roles List Offset: 400
2020/05/13 12:49:42 [DEBUG] Loading User Accounts Roles List Offset: 500
2020/05/13 12:49:43 [DEBUG] Users Roles Loaded: 32
2020/05/13 12:49:43 [DEBUG] Skipping Loading Sites Due to Config
2020/05/13 12:49:43 [DEBUG] Loading Orgs from Hornbill
2020/05/13 12:49:43 [DEBUG] getGroupsList Count: 19
2020/05/13 12:49:43 [DEBUG] Loading Orgs List Offset: 0
2020/05/13 12:49:43 [DEBUG] Orgs Loaded: 19
2020/05/13 12:49:43 [DEBUG] Loading User Orgs from Hornbill
2020/05/13 12:49:43 [DEBUG] Loading User Accounts Orgs List Offset: 0
2020/05/13 12:49:43 [DEBUG] Loading User Accounts Orgs List Offset: 100
2020/05/13 12:49:43 [DEBUG] User Orgs Loaded: 32

{EDIT}

2020/05/13 12:49:46 [MESSAGE] Import Complete
2020/05/13 12:49:46 [ERROR] One or more errors encountered please check the log file
2020/05/13 12:49:46 [ERROR] Error Count: 31
2020/05/13 12:49:46 [MESSAGE] Accounts Procesed: 32
2020/05/13 12:49:46 [MESSAGE] Created: 0
2020/05/13 12:49:46 [MESSAGE] Updated: 1
2020/05/13 12:49:46 [MESSAGE] Status Updates: 0
2020/05/13 12:49:46 [MESSAGE] Profiles Updated: 32
2020/05/13 12:49:46 [MESSAGE] Images Updated: 0
2020/05/13 12:49:46 [MESSAGE] Groups Added: 0
2020/05/13 12:49:46 [MESSAGE] Groups Removed: 0
2020/05/13 12:49:46 [MESSAGE] Roles Added: 0
2020/05/13 12:49:46 [MESSAGE] Time Taken: 1m17s
2020/05/13 12:49:46 [MESSAGE] Total Traffic: 80
2020/05/13 12:49:46 [MESSAGE] ---- XMLMC DB Import Complete ---- 

*NB - errors in the log file seem to relate to existing users - "[ERROR] Unable to Update User: ffffff.llllll@ourdomain.uk Error: There are no values to update"

 

[SamS]

Do the numbers add up? There should be a few entries of

[Azure] Found ### Users

before a

[Azure] No Skip Token Found

signals the end of whatever is found/processed via Azure. It is likely chunked in lots of 100.

The summary at the end is indicative of what has been processed. It currently suggests that from the 100++ Azure accounts found only 32 were processed. That is admittedly quite a disparity.

Looking through the code, I have a sneaky suspicion: Could you please confirm whether the 32 is the last number (or near enough) of users found in Azure before the "No Skip Token Found"-message?

N.B. Errors regarding managers not being found can be ignored - they have no bearing here - as far as I can tell.

[Chris]

Hi @SamS - sorry for the delayed reply.

The log shows the accounts being read in chunks of 100.  The AAD group has a total of 131 members.  

The log shows the following...

2020/05/14 10:13:49 [DEBUG]  [Azure] Determined Token: X%2744537074090001000************************************
2020/05/14 10:13:49 [MESSAGE] [Azure] Found 100 Users
2020/05/14 10:13:49 [MESSAGE] [Azure] Query Azure Data using Graph API. Please wait...

2020/05/14 10:14:01 [DEBUG]  [Azure] No Skip Token Found
2020/05/14 10:14:01 [MESSAGE] [Azure] Found 31 Users
2020/05/14 10:14:01 [DEBUG] Processing DB User Data

 

[SamS]

Hi @Chris,

I believe I might have found and fixed the issue you appear to be having. Please download the latest release (v2.4.2) of the import tool and please get back to me either way.

[Chris]

Hi @SamS - It's working with v2.4.2 :-) !

I've added some more users to the AAD group, run the script, users are created in out Hornbill instance.  Repeated again adding a few more, success again, so the script is pulling in the extra users above the existing 130+ group members.

Many thanks for fixing this so quickly from the time I raised this on the forum.

Chris