Jump to content

Missing users with Azure Import


Recommended Posts

Hi - I have a problem I'm struggling to solve whereby the Azure Import script isn't importing all the members of an AAD group.

  • I've set up the Azure User Import as per the Wiki instructions (https://wiki.hornbill.com/index.php/Azure_User_Import) with this working successfully when the import script (conf.json) reads an AAD group with 20 members. 
  • I've then added another 100 users to the group and re-run the script (many times!) but not all users are imported/created in our Hornbill instance (about 40 were missing). 
  • As part of my trouble shooting I created another AAD group, added the missing users and amended the script to reference this new group instead - on running it these users get imported without any issues. 
  • I reverted the script to the intended AAD group, added 5 extra users to the group and re-ran, but only 1 of the new 5 users get imported/created.

Anyone able to offer any insight or solutions?  Any help really appreciated!

Chris

Link to comment
Share on other sites

Hi @SamS - thanks for your quick reply.  I can't find any clues from the log file.  I'm pasting exerts (with a couple of annotations) below.  Entries have been redacted in places for obvious reasons.

2020/05/13 12:48:28 [DEBUG] ---- XMLMC SQL Import Utility V2.4.1 ----
2020/05/13 12:48:28 [DEBUG] Flag - Config File conf.json
2020/05/13 12:48:28 [DEBUG] Flag - Zone eur
2020/05/13 12:48:28 [DEBUG] Flag - Log Prefix 
2020/05/13 12:48:28 [DEBUG] Flag - Dry Run false
2020/05/13 12:48:28 [DEBUG] Flag - Workers 1
2020/05/13 12:48:28 [MESSAGE] ---- XMLMC DB User Import Utility V2.4.1 ----
2020/05/13 12:48:28 [MESSAGE] Flag - config goAzure2HUserImport
2020/05/13 12:48:28 [MESSAGE] Flag - logprefix 
2020/05/13 12:48:28 [MESSAGE] Flag - dryrun false
2020/05/13 12:48:28 [MESSAGE] Flag - instanceid 
2020/05/13 12:48:28 [MESSAGE] Flag - apikey 
2020/05/13 12:48:28 [MESSAGE] Flag - apitimeout 60
2020/05/13 12:48:28 [MESSAGE] Flag - workers 1

2020/05/13 12:48:28 [MESSAGE] Flag - forcerun false
2020/05/13 12:48:32 [DEBUG] Loading Config File: C:\Hornbill_Import/conf.json
2020/05/13 12:48:33 [DEBUG] Processing Old Log Files Current Retention Set to: 0
2020/05/13 12:48:33 [MESSAGE] Querying Group [*********************]
2020/05/13 12:48:33 [MESSAGE] [Azure] Query Azure Data using Graph API. Please wait...
2020/05/13 12:48:33 [DEBUG] [SCRIPT] Generating Bearer Token
2020/05/13 12:48:33 [DEBUG] [SCRIPT] Got New BearerToken
2020/05/13 12:48:33 [DEBUG] [AZURE] API URL: https://graph.microsoft.com/v1.0/groups/*******************************/members?
2020/05/13 12:48:34 [MESSAGE] [Azure] Connection Successful
2020/05/13 12:48:34 [DEBUG] [SCRIPT] Re-using BearerToken
2020/05/13 12:48:34 [DEBUG] [AZURE] User API URL: https://graph.microsoft.com/v1.0/users/*************************
2020/05/13 12:48:34 [MESSAGE] [Azure] Connection Successful
2020/05/13 12:48:34 [DEBUG] [Azure] Querying Azure Manager. Please wait...

{EDIT} - removed the hundred+ users listed in the log, but the example below is the user that is in the ADD group but does not get imported/created in the Hornbill instance and isn't mentioned again elsewhere in the log


2020/05/13 12:49:18 [DEBUG] [Azure] Querying Azure Manager. Please wait...
2020/05/13 12:49:18 [DEBUG] [SCRIPT] Re-using BearerToken
2020/05/13 12:49:18 [DEBUG] [AZURE] API URL: https://graph.microsoft.com/v1.0/users/Firstname.Lastname@oursmtpdomain.uk/manager
2020/05/13 12:49:19 [DEBUG]  [Azure] Response: No manager found for Firstname.Lastname@oursmtpdomain.uk
2020/05/13 12:49:19 [DEBUG] [SCRIPT] Re-using BearerToken
2020/05/13 12:49:19 [DEBUG] [AZURE] User API URL: https://graph.microsoft.com/v1.0/users/*************************
2020/05/13 12:49:19 [MESSAGE] [Azure] Connection Successful


{EDIT}


2020/05/13 12:49:25 [DEBUG]  [Azure] Determined Token: X%274453707409000100000000000********************************************
2020/05/13 12:49:25 [MESSAGE] [Azure] Found 100 Users
2020/05/13 12:49:25 [MESSAGE] [Azure] Query Azure Data using Graph API. Please wait...
2020/05/13 12:49:25 [DEBUG] [SCRIPT] Re-using BearerToken
2020/05/13 12:49:25 [DEBUG] [AZURE] API URL: https://graph.microsoft.com/v1.0/groups/37ab571d-993b-424c-9e84-f2ee4c3e3893/members?&$skiptoken=X'44537074090001000000000************************************
2020/05/13 12:49:26 [MESSAGE] [Azure] Connection Successful


{EDIT}


2020/05/13 12:49:42 [DEBUG] Loading Users from Hornbill
2020/05/13 12:49:42 [DEBUG] getUserAccountsList Count: 144
2020/05/13 12:49:42 [DEBUG] Loading User Accounts List Offset: 0

2020/05/13 12:49:42 [DEBUG] Loading User Accounts List Offset: 100

2020/05/13 12:49:42 [DEBUG] Users Loaded: 144
2020/05/13 12:49:42 [DEBUG] Loading Users Roles from Hornbill
2020/05/13 12:49:42 [DEBUG] getUserAccountsRolesList Count: 505
2020/05/13 12:49:42 [DEBUG] Loading User Accounts Roles List Offset: 0
2020/05/13 12:49:42 [DEBUG] Loading User Accounts Roles List Offset: 100
2020/05/13 12:49:42 [DEBUG] Loading User Accounts Roles List Offset: 200
2020/05/13 12:49:42 [DEBUG] Loading User Accounts Roles List Offset: 300
2020/05/13 12:49:42 [DEBUG] Loading User Accounts Roles List Offset: 400
2020/05/13 12:49:42 [DEBUG] Loading User Accounts Roles List Offset: 500
2020/05/13 12:49:43 [DEBUG] Users Roles Loaded: 32
2020/05/13 12:49:43 [DEBUG] Skipping Loading Sites Due to Config
2020/05/13 12:49:43 [DEBUG] Loading Orgs from Hornbill
2020/05/13 12:49:43 [DEBUG] getGroupsList Count: 19
2020/05/13 12:49:43 [DEBUG] Loading Orgs List Offset: 0
2020/05/13 12:49:43 [DEBUG] Orgs Loaded: 19
2020/05/13 12:49:43 [DEBUG] Loading User Orgs from Hornbill
2020/05/13 12:49:43 [DEBUG] Loading User Accounts Orgs List Offset: 0
2020/05/13 12:49:43 [DEBUG] Loading User Accounts Orgs List Offset: 100
2020/05/13 12:49:43 [DEBUG] User Orgs Loaded: 32


{EDIT}


2020/05/13 12:49:46 [MESSAGE] Import Complete
2020/05/13 12:49:46 [ERROR] One or more errors encountered please check the log file
2020/05/13 12:49:46 [ERROR] Error Count: 31
2020/05/13 12:49:46 [MESSAGE] Accounts Procesed: 32
2020/05/13 12:49:46 [MESSAGE] Created: 0
2020/05/13 12:49:46 [MESSAGE] Updated: 1
2020/05/13 12:49:46 [MESSAGE] Status Updates: 0
2020/05/13 12:49:46 [MESSAGE] Profiles Updated: 32
2020/05/13 12:49:46 [MESSAGE] Images Updated: 0
2020/05/13 12:49:46 [MESSAGE] Groups Added: 0
2020/05/13 12:49:46 [MESSAGE] Groups Removed: 0
2020/05/13 12:49:46 [MESSAGE] Roles Added: 0
2020/05/13 12:49:46 [MESSAGE] Time Taken: 1m17s
2020/05/13 12:49:46 [MESSAGE] Total Traffic: 80
2020/05/13 12:49:46 [MESSAGE] ---- XMLMC DB Import Complete ---- 

*NB - errors in the log file seem to relate to existing users - "[ERROR] Unable to Update User: ffffff.llllll@ourdomain.uk Error: There are no values to update"


 

Link to comment
Share on other sites

Do the numbers add up? There should be a few entries of

[Azure] Found ### Users

before a

[Azure] No Skip Token Found

signals the end of whatever is found/processed via Azure. It is likely chunked in lots of 100.

The summary at the end is indicative of what has been processed. It currently suggests that from the 100++ Azure accounts found only 32 were processed. That is admittedly quite a disparity.

Looking through the code, I have a sneaky suspicion: Could you please confirm whether the 32 is the last number (or near enough) of users found in Azure before the "No Skip Token Found"-message?

N.B. Errors regarding managers not being found can be ignored - they have no bearing here - as far as I can tell.

Link to comment
Share on other sites

Hi @SamS - sorry for the delayed reply.

The log shows the accounts being read in chunks of 100.  The AAD group has a total of 131 members.  

The log shows the following...

2020/05/14 10:13:49 [DEBUG]  [Azure] Determined Token: X%2744537074090001000************************************
2020/05/14 10:13:49 [MESSAGE] [Azure] Found 100 Users
2020/05/14 10:13:49 [MESSAGE] [Azure] Query Azure Data using Graph API. Please wait...


2020/05/14 10:14:01 [DEBUG]  [Azure] No Skip Token Found
2020/05/14 10:14:01 [MESSAGE] [Azure] Found 31 Users
2020/05/14 10:14:01 [DEBUG] Processing DB User Data

 

Link to comment
Share on other sites

Hi @Chris,

I believe I might have found and fixed the issue you appear to be having. Please download the latest release (v2.4.2) of the import tool and please get back to me either way.

Link to comment
Share on other sites

Hi @SamS - It's working with v2.4.2 :-) !

I've added some more users to the AAD group, run the script, users are created in out Hornbill instance.  Repeated again adding a few more, success again, so the script is pulling in the extra users above the existing 130+ group members.

Many thanks for fixing this so quickly from the time I raised this on the forum.

Chris

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

×
×
  • Create New...