From de91d857d706ac2811607b4938ae0b9ae8d476b3 Mon Sep 17 00:00:00 2001 From: Shachaf Goldstein Date: Thu, 18 Jun 2026 16:49:12 +0300 Subject: [PATCH 1/2] Add Find-CommonGroupsForDestination utility Adds Utils/Find-CommonGroupsForDestination.ps1: given a destination asset (FQDN or IP), find the AD groups shared by the source users that accessed it, to help scope identity-based segmentation rules. Highlights: - Fast source-user collection via /activities/network/distinctField/srcUser with fallback to paging /activities/network for older tenants. - Resolves activity SIDs/userNames to ZN users via SID first, then PrincipalName; dedupes by ZN user id. - Parallel group-membership lookups on PowerShell 7+ (-MaxParallel, default 8, threshold 20 users); sequential on Windows PowerShell 5.1. - Retry + exponential backoff + jitter on transient API failures (HTTP 429 / 5xx); honours Retry-After. - CSV output with optional -IncludeSourceUsers column. Also adds Utils/README.md documenting the new folder and script. --- Utils/Find-CommonGroupsForDestination.ps1 | 499 ++++++++++++++++++++++ Utils/README.md | 122 ++++++ 2 files changed, 621 insertions(+) create mode 100644 Utils/Find-CommonGroupsForDestination.ps1 create mode 100644 Utils/README.md diff --git a/Utils/Find-CommonGroupsForDestination.ps1 b/Utils/Find-CommonGroupsForDestination.ps1 new file mode 100644 index 0000000..183f338 --- /dev/null +++ b/Utils/Find-CommonGroupsForDestination.ps1 @@ -0,0 +1,499 @@ +#Requires -Module ZeroNetworks +<# +.SYNOPSIS + Find common AD groups shared by source users that accessed a specific destination. + +.DESCRIPTION + Queries Zero Networks network activities for a destination asset (by IP or FQDN), + collects distinct source users, retrieves all their group memberships via the ZN + module, and reports every group shared by at least MinUserCount of those users. + Output is a CSV with one row per qualifying group. + + AUTHENTICATION (required before running): + The script uses the API key stored in the $env:ZNApiKey environment variable. + Set it using either of: + * Set-ZNApiKey -ApiKey '' - stores the key in $env:ZNApiKey for the session. + * Connect-ZN -Email '' - interactive OTP login that populates $env:ZNApiKey. + * Or set the variable directly: $env:ZNApiKey = '' + If $env:ZNApiKey is not set, the script stops with an authentication error. + +.PARAMETER DestinationFQDN + FQDN of the destination asset (mutually exclusive with DestinationIP). + +.PARAMETER DestinationIP + IPv4 address of the destination asset (mutually exclusive with DestinationFQDN). + +.PARAMETER MinUserCount + Minimum number of source users that must share a group for it to appear in the + report. Default: 2. + +.PARAMETER FromDays + How many days back to search for activities. Default: 30. + +.PARAMETER OutputPath + Full path for the output CSV file. If omitted, a timestamped file is created in + the current directory. + +.PARAMETER IncludeSourceUsers + Off by default. When specified, adds a 'SourceUsers' column listing the member + names in each group. Omitted by default because it becomes very noisy when groups + contain thousands of users; the 'SourceUserCount' column is always present. + +.PARAMETER MaxParallel + Max concurrent group-membership lookups (1-16, default 8). Parallelism is used on + PowerShell 7+; on Windows PowerShell 5.1 lookups run sequentially regardless. + Lower this if the tenant returns HTTP 429 (rate limiting). All API calls already + retry with exponential backoff. + +.EXAMPLE + .\Find-CommonGroupsForDestination.ps1 -DestinationFQDN "fileserver.corp.local" -MinUserCount 3 -FromDays 7 + +.EXAMPLE + .\Find-CommonGroupsForDestination.ps1 -DestinationIP "10.10.0.50" -MinUserCount 2 -OutputPath "C:\Reports\groups.csv" +#> + +[CmdletBinding(DefaultParameterSetName = 'Help')] +param( + [Parameter(Mandatory, ParameterSetName = 'FQDN')] + [string]$DestinationFQDN, + + [Parameter(Mandatory, ParameterSetName = 'IP')] + [string]$DestinationIP, + + [Parameter()] + [ValidateRange(1, [int]::MaxValue)] + [int]$MinUserCount = 2, + + [Parameter()] + [ValidateRange(1, 365)] + [int]$FromDays = 30, + + [Parameter()] + [string]$OutputPath, + + [Parameter()] + [switch]$IncludeSourceUsers, + + [Parameter()] + [ValidateRange(1, 16)] + [int]$MaxParallel = 8 +) + +Set-StrictMode -Version Latest +$ErrorActionPreference = 'Stop' + +# --------------------------------------------------------------------------- +# 0. No destination supplied -> show usage instead of a parameter-set error. +# --------------------------------------------------------------------------- +if ($PSCmdlet.ParameterSetName -eq 'Help') { + Write-Host @" + +Find-CommonGroupsForDestination.ps1 + Finds AD groups shared by the source users that accessed a given destination, + and writes the common groups (>= a threshold) to a CSV. + +USAGE + .\Find-CommonGroupsForDestination.ps1 -DestinationFQDN [options] + .\Find-CommonGroupsForDestination.ps1 -DestinationIP [options] + +REQUIRED (choose one) + -DestinationFQDN FQDN of the destination (managed asset or external domain). + -DestinationIP IPv4 address of the destination. + +OPTIONS + -MinUserCount Min source users sharing a group to report it. Default: 2 + -FromDays Days of activity history to scan (1-365). Default: 30 + -OutputPath CSV output path. Default: .\CommonGroups__.csv + -IncludeSourceUsers Add a 'SourceUsers' column listing member names. Default: off (noisy at scale) + +PREREQUISITE (authentication required) + The script reads the API key from `$env:ZNApiKey`. Set it first using either: + Set-ZNApiKey -ApiKey '' # stores key in `$env:ZNApiKey` for the session + Connect-ZN -Email '' # interactive OTP login, populates `$env:ZNApiKey` + `$env:ZNApiKey = '' # or set the env var directly + +EXAMPLES + .\Find-CommonGroupsForDestination.ps1 -DestinationFQDN "fileserver.corp.local" -MinUserCount 3 -FromDays 7 + .\Find-CommonGroupsForDestination.ps1 -DestinationIP "10.10.0.50" + + For full help: Get-Help .\Find-CommonGroupsForDestination.ps1 -Full + +"@ -ForegroundColor Cyan + return +} + +# Make -Debug non-interactive (default behaviour would prompt on every Write-Debug). +if ($PSBoundParameters.ContainsKey('Debug')) { $DebugPreference = 'Continue' } + +# Safe accessor for optional JSON properties (StrictMode throws on missing ones). +function Get-Prop { + param($Object, [string]$Name) + if ($null -eq $Object) { return $null } + $p = $Object.PSObject.Properties[$Name] + if ($p) { return $p.Value } + return $null +} + +# Run a scriptblock with retry + exponential backoff on transient API failures +# (HTTP 429 / 5xx and transport-level errors). Non-transient errors (e.g. 400/401/ +# 403/404) are rethrown immediately. Honors a Retry-After header when present. +function Invoke-WithRetry { + param( + [Parameter(Mandatory)][scriptblock]$Action, + [string]$Description = 'API call', + [int]$MaxAttempts = 4, + [int]$BaseDelayMs = 1000 + ) + for ($attempt = 1; ; $attempt++) { + try { + return & $Action + } catch { + $status = $null + try { $status = [int]$_.Exception.Response.StatusCode } catch {} + $transient = ($status -in 429, 500, 502, 503, 504) -or ($null -eq $status) + if (-not $transient -or $attempt -ge $MaxAttempts) { throw } + + $delayMs = $BaseDelayMs * [math]::Pow(2, $attempt - 1) + try { + $ra = $_.Exception.Response.Headers.RetryAfter.Delta.TotalMilliseconds + if ($ra) { $delayMs = [math]::Max($delayMs, $ra) } + } catch {} + $delayMs = [int]($delayMs + (Get-Random -Minimum 0 -Maximum 250)) # jitter + Write-Verbose "$Description failed (attempt $attempt/$MaxAttempts, status=$status); retrying in $delayMs ms." + Start-Sleep -Milliseconds $delayMs + } + } +} + +# --------------------------------------------------------------------------- +# 1. Validate authentication +# --------------------------------------------------------------------------- +if (-not $env:ZNApiKey) { + throw "Not authenticated: `$env:ZNApiKey is not set. Run Set-ZNApiKey -ApiKey '' or Connect-ZN (or set `$env:ZNApiKey directly) before running this script." +} + +$accountName = (Read-ZNJWTtoken $env:ZNApiKey).aud.split('.')[0] +$baseUrl = "https://$accountName.zeronetworks.com/api/v1" +$headers = @{ Authorization = $env:ZNApiKey } +Write-Verbose "Authenticated to account '$accountName' (base URL: $baseUrl)." + +$epoch = [datetime]'1970-01-01T00:00:00Z' +$fromEpochMs = [int64][math]::Floor(((Get-Date).AddDays(-$FromDays).ToUniversalTime() - $epoch).TotalMilliseconds) +$toEpochMs = [int64][math]::Floor(((Get-Date).ToUniversalTime() - $epoch).TotalMilliseconds) +Write-Verbose ("Time window: {0:u} .. {1:u} ({2} days)" -f (Get-Date).AddDays(-$FromDays).ToUniversalTime(), (Get-Date).ToUniversalTime(), $FromDays) +Write-Debug "Epoch window (ms): from=$fromEpochMs to=$toEpochMs" + +if (-not $OutputPath) { + $slug = if ($DestinationFQDN) { + $DestinationFQDN -replace '[^\w.]', '_' + } else { + $DestinationIP -replace '\.', '_' + } + $OutputPath = ".\CommonGroups_${slug}_$(Get-Date -Format 'yyyyMMdd_HHmmss').csv" +} +Write-Verbose "Output CSV path: $OutputPath" + +# --------------------------------------------------------------------------- +# 2. Build the server-side destination filter +# +# The /activities/network feed is global; we scope it with the `_filters` +# query parameter (a JSON array of {id, includeValues}). The `dstAsset` +# filter does a partial match on the destination asset name OR external +# domain, so it covers both managed assets and internet FQDNs; `dstIpAddress` +# matches an exact IP or CIDR. Because dstAsset is a partial match, we keep a +# light exact post-filter on the nested `dst` object for precision. +# --------------------------------------------------------------------------- +$destLabel = if ($DestinationFQDN) { $DestinationFQDN } else { $DestinationIP } +Write-Host "Destination: $destLabel" -ForegroundColor Cyan + +if ($PSCmdlet.ParameterSetName -eq 'FQDN') { + $filterId = 'dstAsset' + $filterVal = $DestinationFQDN + $dstMatches = { param($dst) (Get-Prop $dst 'fqdn') -eq $DestinationFQDN } +} else { + $filterId = 'dstIpAddress' + $filterVal = $DestinationIP + $dstMatches = { param($dst) (Get-Prop $dst 'ip') -eq $DestinationIP } +} + +$filtersJson = ConvertTo-Json @([ordered]@{ id = $filterId; includeValues = @($filterVal) }) -Depth 5 -Compress +$filtersEnc = [uri]::EscapeDataString($filtersJson) +Write-Host " Filter: $filtersJson" -ForegroundColor DarkGray +Write-Host " Searching last $FromDays days of network activities..." -ForegroundColor DarkGray +Write-Verbose "Destination filter id='$filterId', value='$filterVal'." +Write-Debug "Encoded _filters: $filtersEnc" + +# --------------------------------------------------------------------------- +# 3. Collect the distinct source users that reached the destination. +# +# Primary path: the distinctField endpoint returns the de-duplicated list of +# source users (with per-user hit counts) for the destination filter in ONE +# call -- far cheaper than paging every activity record (which can be tens of +# thousands of rows on a busy destination). If that endpoint is unavailable +# (older tenant), we fall back to paging /activities/network. +# +# distinctField returns only the userName (DOMAIN\user); activity records +# also carry the SID. Either way we resolve to a ZN user id in step 4. +# --------------------------------------------------------------------------- +$sourceUsers = [ordered]@{} # key -> @{ Sid; Name; Count } + +function Add-SourceUser { + param([string]$Sid, [string]$Name, [int]$Count = 0) + if (-not $Name -and -not $Sid) { return } + $key = if ($Sid) { "sid:$Sid" } else { "name:$($Name.ToLowerInvariant())" } + if (-not $sourceUsers.Contains($key)) { + $sourceUsers[$key] = @{ Sid = $Sid; Name = $Name; Count = $Count } + } +} + +$usedDistinctField = $false +$distinctUrl = "$baseUrl/activities/network/distinctField/srcUser?from=$fromEpochMs&to=$toEpochMs&_filters=$filtersEnc" +Write-Debug "GET $distinctUrl" +try { + $df = Invoke-WithRetry -Description 'distinctField/srcUser' -Action { + Invoke-RestMethod -Uri $distinctUrl -Headers $headers -Verbose:$false -Debug:$false + } + $aggs = @(Get-Prop (Get-Prop $df 'items') 'aggregations') + foreach ($a in $aggs) { + $nm = Get-Prop $a 'name' + if (-not $nm -or $nm -eq 'Unknown') { continue } + Add-SourceUser -Name $nm -Count ([int](Get-Prop $a 'count')) + Write-Debug "distinct srcUser: '$nm' (count=$(Get-Prop $a 'count'))" + } + $usedDistinctField = $true + Write-Host " Distinct source users (summarized): $($sourceUsers.Count)" -ForegroundColor DarkGray + Write-Verbose "Used distinctField/srcUser endpoint: 1 call, $($aggs.Count) raw values -> $($sourceUsers.Count) distinct." +} catch { + Write-Verbose "distinctField endpoint unavailable ($($_.Exception.Message)); falling back to full activity scan." +} + +if (-not $usedDistinctField) { + # Fallback: page the full activity feed and dedupe source users client-side. + $cursor = $null; $pageSize = 400; $actUrl = "$baseUrl/activities/network" + $totalFetched = 0; $matchedCount = 0; $pageNum = 0 + do { + $pageNum++ + $query = "?_limit=$pageSize&from=$fromEpochMs&to=$toEpochMs&_filters=$filtersEnc" + if ($cursor) { $query += "&_cursor=$cursor" } + Write-Debug "GET $actUrl$query" + try { + $page = Invoke-WithRetry -Description "activities page $pageNum" -Action { + Invoke-RestMethod -Uri "$actUrl$query" -Headers $headers -Verbose:$false -Debug:$false + } + } catch { + Write-Warning "Activities API call failed after retries: $_" + break + } + $items = @($page.items) + $totalFetched += $items.Count + foreach ($act in $items) { + # Server already filtered by destination; confirm exact dst match. + if (-not (& $dstMatches (Get-Prop $act 'dst'))) { continue } + $matchedCount++ + $src = Get-Prop $act 'src' + $sid = Get-Prop $src 'userId' + if ($sid) { Add-SourceUser -Sid $sid -Name (Get-Prop $src 'userName') } + } + $cursor = $page.scrollCursor + Write-Progress -Activity "Scanning network activities for $destLabel" ` + -Status ("Page $pageNum | scanned $totalFetched | $($sourceUsers.Count) distinct source users") + Write-Verbose ("Page {0}: scanned={1} matched={2} distinctUsers={3}" -f $pageNum, $totalFetched, $matchedCount, $sourceUsers.Count) + } while ($items.Count -eq $pageSize -and $cursor) + Write-Progress -Activity "Scanning network activities for $destLabel" -Completed + Write-Host " Total activities scanned : $totalFetched" -ForegroundColor DarkGray + Write-Host " Distinct source users : $($sourceUsers.Count)" -ForegroundColor DarkGray +} + +if ($sourceUsers.Count -eq 0) { + Write-Warning "No source users found for '$destLabel' in the last $FromDays days." + return +} + +# --------------------------------------------------------------------------- +# 3b. Build a SID/PrincipalName -> ZN user lookup from the directory +# +# Activity SIDs do not always match user.Sid (e.g. Entra synthetic SIDs), +# so we resolve by SID first, then fall back to src.userName == PrincipalName. +# --------------------------------------------------------------------------- +Write-Host "`nBuilding user directory map..." -ForegroundColor Cyan + +$bySid = @{}; $byPrincipal = @{} +$offset = 0 +do { + Write-Debug "Get-ZNUser -Limit 400 -Offset $offset" + $offsetLocal = $offset + $uPage = Invoke-WithRetry -Description "Get-ZNUser offset $offsetLocal" -Action { + Get-ZNUser -Limit 400 -Offset $offsetLocal -Verbose:$false -Debug:$false + } + $uItems = @($uPage.Items) + foreach ($u in $uItems) { + if ($u.Sid) { $bySid[$u.Sid] = $u } + if ($u.PrincipalName) { $byPrincipal[$u.PrincipalName.ToLowerInvariant()] = $u } + } + $offset += $uItems.Count + Write-Verbose "Directory page fetched: $($uItems.Count) users (total indexed: $offset)." +} while ($uItems.Count -eq 400) + +Write-Host " Directory users indexed : $($bySid.Count) by SID, $($byPrincipal.Count) by principal name" -ForegroundColor DarkGray + +# --------------------------------------------------------------------------- +# 4a. Resolve each distinct source user to a unique ZN user id. +# (SID first, then userName == PrincipalName; dedupe by ZN id.) +# --------------------------------------------------------------------------- +Write-Host "`nResolving users..." -ForegroundColor Cyan + +$resolvedUsers = [System.Collections.Generic.List[object]]::new() # @{ Id; Name } +$unresolved = [System.Collections.Generic.List[string]]::new() +$processedIds = [System.Collections.Generic.HashSet[string]]::new() + +foreach ($kvp in $sourceUsers.GetEnumerator()) { + $src = $kvp.Value + $userName = if ($src.Name) { $src.Name } else { $src.Sid } + + $znUser = $null; $resolvedVia = $null + if ($src.Sid -and $bySid.ContainsKey($src.Sid)) { + $znUser = $bySid[$src.Sid]; $resolvedVia = 'SID' + } elseif ($src.Name -and $byPrincipal.ContainsKey($src.Name.ToLowerInvariant())) { + $znUser = $byPrincipal[$src.Name.ToLowerInvariant()]; $resolvedVia = 'PrincipalName' + } + + if (-not $znUser) { + # Typically well-known/system/machine accounts (SYSTEM, LOCAL SERVICE, *$) - no AD groups. + $unresolved.Add($userName) + Write-Verbose "Unresolved source user '$userName' (sid='$($src.Sid)') - skipped." + continue + } + # Case-variant userNames / multiple SIDs can map to one ZN user; count once. + if (-not $processedIds.Add($znUser.Id)) { + Write-Debug "Duplicate -> ZN user $($znUser.Id) already processed; skipping '$userName'." + continue + } + $resolvedUsers.Add([pscustomobject]@{ Id = $znUser.Id; Name = $znUser.Name }) + Write-Debug "Resolved '$userName' via $resolvedVia -> ZN user $($znUser.Id)" +} + +Write-Host " Source users resolved : $($resolvedUsers.Count) / $($sourceUsers.Count)" -ForegroundColor DarkGray +if ($unresolved.Count -gt 0) { + Write-Host " Unresolved (skipped) : $($unresolved.Count) (e.g. $((($unresolved | Select-Object -First 3) -join ', ')))" -ForegroundColor DarkGray +} + +# --------------------------------------------------------------------------- +# 4b. Retrieve each resolved user's group memberships. +# Parallel on PowerShell 7+ (capped by -MaxParallel); sequential on 5.1. +# Each call is retry-wrapped. Returns @{ Name; Groups }. +# --------------------------------------------------------------------------- +Write-Host "Retrieving group memberships for $($resolvedUsers.Count) user(s)..." -ForegroundColor Cyan + +# Parallelism only pays off once there are enough users to amortize runspace +# startup cost; below the threshold, sequential is faster. Tune via -MaxParallel. +$parallelThreshold = 20 +$useParallel = ($PSVersionTable.PSVersion.Major -ge 7) -and ($MaxParallel -gt 1) -and ($resolvedUsers.Count -ge $parallelThreshold) +Write-Verbose ("Membership lookup mode: {0} ({1} users, PS {2})." -f ` + $(if ($useParallel) {"parallel, throttle $MaxParallel"} else {"sequential (< $parallelThreshold users or PS<7 or -MaxParallel 1)"}), ` + $resolvedUsers.Count, $PSVersionTable.PSVersion) + +if ($useParallel) { + # Call the REST endpoint that Get-ZNUserMemberOf wraps (GET /users/{id}/ancestors) + # directly, so parallel runspaces do NOT each pay the (heavy) Import-Module cost. + $apiKey = $env:ZNApiKey + $base = $baseUrl + $membershipResults = $resolvedUsers | ForEach-Object -ThrottleLimit $MaxParallel -Parallel { + $u = $_ + $hdr = @{ Authorization = $using:apiKey } + $uri = "$using:base/users/$($u.Id)/ancestors" + $groups = $null + for ($attempt = 1; $attempt -le 4; $attempt++) { + try { + $groups = (Invoke-RestMethod -Uri $uri -Headers $hdr -Verbose:$false).items + break + } catch { + $status = $null; try { $status = [int]$_.Exception.Response.StatusCode } catch {} + $transient = ($status -in 429,500,502,503,504) -or ($null -eq $status) + if (-not $transient -or $attempt -eq 4) { + Write-Warning " Could not retrieve groups for $($u.Name) ($($u.Id)): $_" + break + } + Start-Sleep -Milliseconds ([int](1000 * [math]::Pow(2, $attempt-1) + (Get-Random -Maximum 250))) + } + } + [pscustomobject]@{ Name = $u.Name; Groups = @($groups) } + } +} else { + $membershipResults = foreach ($u in $resolvedUsers) { + $groups = $null + try { + $groups = Invoke-WithRetry -Description "Get-ZNUserMemberOf $($u.Id)" -Action { + (Get-ZNUserMemberOf -UserId $u.Id -Verbose:$false -Debug:$false).Items + } + } catch { + Write-Warning " Could not retrieve groups for $($u.Name) ($($u.Id)): $_" + } + Write-Verbose "User '$($u.Name)' ($($u.Id)) is a member of $(@($groups).Count) group(s)." + [pscustomobject]@{ Name = $u.Name; Groups = @($groups) } + } +} + +# --------------------------------------------------------------------------- +# 4c. Merge memberships into the group -> users map (sequential; no races). +# --------------------------------------------------------------------------- +# groupId -> @{ Name; Id; Users (list of display names) } +$groupUserMap = [ordered]@{} +foreach ($res in $membershipResults) { + foreach ($g in $res.Groups) { + if (-not $g) { continue } + $gid = $g.Id + if (-not $groupUserMap.Contains($gid)) { + $groupUserMap[$gid] = @{ + Name = $g.Name + Id = $gid + Users = [System.Collections.Generic.List[string]]::new() + } + } + $groupUserMap[$gid].Users.Add($res.Name) + } +} + +# --------------------------------------------------------------------------- +# 5. Filter groups by threshold and build report +# --------------------------------------------------------------------------- +Write-Verbose "Distinct groups seen across resolved users: $($groupUserMap.Count). Applying threshold >= $MinUserCount." + +$qualifying = $groupUserMap.Values | + Where-Object { $_.Users.Count -ge $MinUserCount } | + Sort-Object { $_.Users.Count } -Descending + +if ($qualifying.Count -eq 0) { + Write-Warning "No groups found with $MinUserCount or more source users. Try lowering -MinUserCount." + return +} + +Write-Host "`n$($qualifying.Count) group(s) with >= $MinUserCount source users." -ForegroundColor Green + +if ($IncludeSourceUsers) { + Write-Verbose "Including 'SourceUsers' column (member names) in output." +} else { + Write-Verbose "Omitting member names; use -IncludeSourceUsers to add the 'SourceUsers' column." +} + +$csvRows = foreach ($g in $qualifying) { + $row = [ordered]@{ + GroupName = $g.Name + GroupId = $g.Id + SourceUserCount = $g.Users.Count + } + if ($IncludeSourceUsers) { + $row.SourceUsers = ($g.Users | Sort-Object) -join '; ' + } + $row.Destination = $destLabel + $row.LookbackDays = $FromDays + [PSCustomObject]$row +} + +$csvRows | Export-Csv -Path $OutputPath -NoTypeInformation -Encoding UTF8 +Write-Verbose "Wrote $($csvRows.Count) row(s) to $OutputPath." + +Write-Host "Report saved : $OutputPath" -ForegroundColor Green +Write-Host "" +Write-Host "=== Top 10 Common Groups ===" -ForegroundColor Cyan +$topCols = if ($IncludeSourceUsers) { 'GroupName','SourceUserCount','SourceUsers' } else { 'GroupName','SourceUserCount' } +$csvRows | Select-Object -First 10 | Format-Table $topCols -AutoSize -Wrap diff --git a/Utils/README.md b/Utils/README.md new file mode 100644 index 0000000..2c0094a --- /dev/null +++ b/Utils/README.md @@ -0,0 +1,122 @@ +# Utils + +Standalone utility scripts that complement the Zero Networks platform but don't fit cleanly under `Segment/`, `Connect/`, `TrustMeter/`, or `Detections/`. + +## Contents + +| Script | Purpose | +|---|---| +| [Find-CommonGroupsForDestination.ps1](Find-CommonGroupsForDestination.ps1) | Find the AD groups shared by users that accessed a given destination, to help build identity-based segmentation rules. | + +--- + +## Find-CommonGroupsForDestination.ps1 + +Queries Zero Networks network activities for a destination asset (by IP or FQDN), collects the distinct source users that contacted it, retrieves each user's group memberships from the ZN directory, and writes a CSV listing every AD group shared by at least `MinUserCount` of those users. + +The output is intended to help answer: *"Which AD group(s) should I scope this destination's access rule to?"* + +### Requirements + +- **PowerShell 5.1+** (Windows PowerShell or PowerShell 7) +- **[ZeroNetworks PowerShell module](https://www.powershellgallery.com/packages/ZeroNetworks)** (`#Requires -Module ZeroNetworks`) +- **Zero Networks API key** with permission to read network activities and the user directory + +### Authentication + +The script reads the API key from `$env:ZNApiKey`. Set it once per session using any of: + +```powershell +Set-ZNApiKey -ApiKey '' # stores key in $env:ZNApiKey for the session +Connect-ZN -Email '' # interactive OTP login, populates $env:ZNApiKey +$env:ZNApiKey = '' # or set the env var directly +``` + +If `$env:ZNApiKey` is not set the script exits with an authentication error. + +### Parameters + +| Parameter | Required | Default | Description | +|---|---|---|---| +| `-DestinationFQDN` | one of these | – | FQDN of the destination asset (managed asset or external domain). | +| `-DestinationIP` | one of these | – | IPv4 address of the destination asset. | +| `-MinUserCount` | no | `2` | Minimum number of source users that must share a group for it to appear in the report. | +| `-FromDays` | no | `30` | How many days of activity history to scan (`1`–`365`). | +| `-OutputPath` | no | `./CommonGroups__.csv` | Output CSV path. | +| `-IncludeSourceUsers` | no | off | Switch. Adds a `SourceUsers` column to the CSV listing the member display names per group. Off by default — groups containing thousands of users make the column very noisy. `SourceUserCount` is always present. | +| `-MaxParallel` | no | `8` | Max concurrent group-membership lookups (`1`–`16`). Used only on PowerShell 7+ and only once there are at least 20 resolved users to amortize runspace startup. Set to `1` to force sequential, lower if the tenant returns HTTP 429. Ignored on Windows PowerShell 5.1. | + +`-DestinationFQDN` and `-DestinationIP` are mutually exclusive — supply exactly one. Running the script with neither prints usage and exits. + +`-Verbose` and `-Debug` are honoured (`-Debug` is forced non-interactive). API tracing inside the ZN module is suppressed so the key isn't echoed to the console. + +### How it works + +1. Validates `$env:ZNApiKey` and decodes the JWT to derive the tenant base URL. +2. Builds a `_filters` query (`dstAsset` for FQDN, `dstIpAddress` for IP) scoped to the requested time window. +3. **Collects distinct source users** in one of two ways: + - **Fast path (primary):** calls `/activities/network/distinctField/srcUser`, which returns the de-duplicated set of source users (with per-user hit counts) in a single request. This avoids paging through tens of thousands of activity rows on busy destinations. + - **Fallback:** if the `distinctField` endpoint is unavailable (older tenant), the script pages `/activities/network` and de-duplicates client-side. +4. Pages through `Get-ZNUser` to build a SID → user and PrincipalName → user lookup, so activity SIDs that don't match ZN user IDs (e.g. Entra synthetic SIDs) can still be resolved. +5. **Resolves** each distinct source user against that lookup (SID first, then PrincipalName). If two source entries map to the same ZN user (case-variant `userName`, or multiple SIDs for one identity) the user is counted **once**. Unresolvable system/machine accounts are skipped and reported. +6. **Retrieves group memberships** for each resolved user: + - On **PowerShell 7+** with ≥ 20 resolved users, the script fans out to `-MaxParallel` concurrent runspaces (default 8) calling `/users/{id}/ancestors` directly — this avoids the per-runspace cost of re-importing the ZN module. + - On **Windows PowerShell 5.1**, on PS 7+ with fewer than 20 resolved users, or when `-MaxParallel 1` is supplied, the script runs sequentially via `Get-ZNUserMemberOf`. + - All API calls (distinctField, activity pages, `Get-ZNUser`, membership lookups) are wrapped with retry + exponential backoff + jitter on transient failures (HTTP 429 / 5xx and transport errors). `Retry-After` headers are honoured. +7. Merges memberships into a group → users map (sequential, race-free), filters out groups below `-MinUserCount`, sorts by member count desc, writes the CSV, and prints the top 10 to the console. + +### Output + +The script writes a CSV with one row per qualifying group: + +| Column | Always present? | Description | +|---|---|---| +| `GroupName` | yes | Display name of the AD group. | +| `GroupId` | yes | Zero Networks group ID. | +| `SourceUserCount` | yes | Number of source users (that accessed the destination) who are members. | +| `SourceUsers` | only with `-IncludeSourceUsers` | Semicolon-separated list of those user display names. | +| `Destination` | yes | The destination FQDN or IP that was queried. | +| `LookbackDays` | yes | The `-FromDays` value used for the run. | + +A summary table (top 10 by `SourceUserCount`) is also printed to the console — it includes the `SourceUsers` column only when `-IncludeSourceUsers` is set. + +### Usage + +```powershell +# Authenticate once +Set-ZNApiKey -ApiKey '' + +# Find groups shared by >= 3 users that hit a file server in the last 7 days +.\Find-CommonGroupsForDestination.ps1 ` + -DestinationFQDN 'fileserver.corp.local' ` + -MinUserCount 3 ` + -FromDays 7 + +# Same, by IP, with an explicit output path +.\Find-CommonGroupsForDestination.ps1 ` + -DestinationIP '10.10.0.50' ` + -OutputPath 'C:\Reports\fileserver-groups.csv' + +# Include the SourceUsers column (member display names per group) +.\Find-CommonGroupsForDestination.ps1 ` + -DestinationFQDN 'fileserver.corp.local' ` + -IncludeSourceUsers + +# Throttle parallel membership lookups (e.g. tenant is rate-limiting on PS 7+) +.\Find-CommonGroupsForDestination.ps1 ` + -DestinationFQDN 'fileserver.corp.local' ` + -MaxParallel 2 + +# See the full PowerShell help (synopsis, parameters, examples) +Get-Help .\Find-CommonGroupsForDestination.ps1 -Full +``` + +### Notes & caveats + +- The fast path (`distinctField/srcUser`) returns only `userName` (`DOMAIN\user`), so users are resolved by `PrincipalName`. The fallback activity-scan path also captures the SID and tries SID first. Either way, well-known/system/machine accounts (`SYSTEM`, `LOCAL SERVICE`, `*$`, etc.) won't resolve to a ZN user and are skipped — the count of unresolved users is reported. +- Only activities that include source-user information are considered. Service-to-service or agentless flows without a resolvable user contribute nothing to the report. +- If two source entries map to the same ZN user, the user is counted **once** per group (no double-counting from SID/name variants). +- The activity-feed fallback path uses cursor pagination at `_limit=400`; very long lookbacks on busy destinations will take longer and consume more API calls. The fast path is unaffected. +- Lower `-MinUserCount` to surface narrower groups, raise it to focus on broadly shared groups. +- `-IncludeSourceUsers` is off by default because the column is unbounded — a group of 10,000 users yields a single 10,000-entry semicolon-joined string per row. Turn it on for small groups or targeted investigations. +- If the tenant returns HTTP 429 (rate limiting), the built-in retry/backoff usually handles it transparently; if it persists, lower `-MaxParallel` (or set it to `1`). From e273e673a54db116cf66704b5c8c9a598ca46f02 Mon Sep 17 00:00:00 2001 From: Shachaf Goldstein Date: Thu, 18 Jun 2026 17:40:58 +0300 Subject: [PATCH 2/2] Resolve source users on demand instead of pre-fetching the directory Replaces the upfront Get-ZNUser pagination with per-user resolves via GET /users/searchIdByPrincipalName?principalName=DOMAIN\user (and searchIdBySid for the activity-scan fallback). This scales to tenants with 100k+ users where the directory pre-fetch would be slow and mostly wasted work. HTTP 404 from the resolver is treated as "not a ZN user" (system/machine/unknown) and skipped silently. Other changes: - Fuse resolve + group-membership fetch into one per-user unit of work, so each parallel runspace does both API calls. - Raise -MaxParallel ceiling from 16 to 64 for very large destinations (default still 8, threshold still 20 source users). - Streaming progress with live ETA / rate as each result lands. - 404 short-circuits the retry loop (only transient errors retried). Updates Utils/README.md to match. --- Utils/Find-CommonGroupsForDestination.ps1 | 257 ++++++++++++---------- Utils/README.md | 22 +- 2 files changed, 156 insertions(+), 123 deletions(-) diff --git a/Utils/Find-CommonGroupsForDestination.ps1 b/Utils/Find-CommonGroupsForDestination.ps1 index 183f338..45ea55f 100644 --- a/Utils/Find-CommonGroupsForDestination.ps1 +++ b/Utils/Find-CommonGroupsForDestination.ps1 @@ -40,10 +40,11 @@ contain thousands of users; the 'SourceUserCount' column is always present. .PARAMETER MaxParallel - Max concurrent group-membership lookups (1-16, default 8). Parallelism is used on - PowerShell 7+; on Windows PowerShell 5.1 lookups run sequentially regardless. - Lower this if the tenant returns HTTP 429 (rate limiting). All API calls already - retry with exponential backoff. + Max concurrent user-resolution / group-membership lookups (1-64, default 8). + Parallelism is used on PowerShell 7+ once there are >= 20 source users; on Windows + PowerShell 5.1 lookups always run sequentially. Raise it (e.g. 24-32) for very large + destinations with thousands of source users; lower it (or set 1) if the tenant + returns HTTP 429 (rate limiting). All API calls already retry with exponential backoff. .EXAMPLE .\Find-CommonGroupsForDestination.ps1 -DestinationFQDN "fileserver.corp.local" -MinUserCount 3 -FromDays 7 @@ -75,7 +76,7 @@ param( [switch]$IncludeSourceUsers, [Parameter()] - [ValidateRange(1, 16)] + [ValidateRange(1, 64)] [int]$MaxParallel = 8 ) @@ -105,6 +106,7 @@ OPTIONS -FromDays Days of activity history to scan (1-365). Default: 30 -OutputPath CSV output path. Default: .\CommonGroups__.csv -IncludeSourceUsers Add a 'SourceUsers' column listing member names. Default: off (noisy at scale) + -MaxParallel Concurrent lookups on PS7+ (1-64). Default: 8 (raise for huge dests; 1 = sequential) PREREQUISITE (authentication required) The script reads the API key from `$env:ZNApiKey`. Set it first using either: @@ -310,149 +312,176 @@ if ($sourceUsers.Count -eq 0) { } # --------------------------------------------------------------------------- -# 3b. Build a SID/PrincipalName -> ZN user lookup from the directory +# 4. Resolve each distinct source user to a ZN user id and fetch its groups. # -# Activity SIDs do not always match user.Sid (e.g. Entra synthetic SIDs), -# so we resolve by SID first, then fall back to src.userName == PrincipalName. -# --------------------------------------------------------------------------- -Write-Host "`nBuilding user directory map..." -ForegroundColor Cyan - -$bySid = @{}; $byPrincipal = @{} -$offset = 0 -do { - Write-Debug "Get-ZNUser -Limit 400 -Offset $offset" - $offsetLocal = $offset - $uPage = Invoke-WithRetry -Description "Get-ZNUser offset $offsetLocal" -Action { - Get-ZNUser -Limit 400 -Offset $offsetLocal -Verbose:$false -Debug:$false - } - $uItems = @($uPage.Items) - foreach ($u in $uItems) { - if ($u.Sid) { $bySid[$u.Sid] = $u } - if ($u.PrincipalName) { $byPrincipal[$u.PrincipalName.ToLowerInvariant()] = $u } - } - $offset += $uItems.Count - Write-Verbose "Directory page fetched: $($uItems.Count) users (total indexed: $offset)." -} while ($uItems.Count -eq 400) - -Write-Host " Directory users indexed : $($bySid.Count) by SID, $($byPrincipal.Count) by principal name" -ForegroundColor DarkGray - -# --------------------------------------------------------------------------- -# 4a. Resolve each distinct source user to a unique ZN user id. -# (SID first, then userName == PrincipalName; dedupe by ZN id.) +# We do NOT enumerate the whole directory (a large tenant can have 100k+ +# users). Instead each distinct source user is resolved on demand: +# name -> GET /users/searchIdByPrincipalName?principalName= +# sid -> GET /users/searchIdBySid?sid= (activity-scan fallback only) +# HTTP 404 = not a ZN user (system/machine/unknown account) -> skipped. +# Then GET /users/{id}/ancestors returns that user's groups. +# +# Runs in parallel on PowerShell 7+ (capped by -MaxParallel) once there are +# enough users to amortize runspace startup; sequential otherwise. Each call +# is retry-wrapped. Case-variant names / multiple SIDs can resolve to the same +# ZN id -> deduped at merge so each user is counted once. # --------------------------------------------------------------------------- -Write-Host "`nResolving users..." -ForegroundColor Cyan - -$resolvedUsers = [System.Collections.Generic.List[object]]::new() # @{ Id; Name } -$unresolved = [System.Collections.Generic.List[string]]::new() -$processedIds = [System.Collections.Generic.HashSet[string]]::new() - -foreach ($kvp in $sourceUsers.GetEnumerator()) { - $src = $kvp.Value - $userName = if ($src.Name) { $src.Name } else { $src.Sid } - - $znUser = $null; $resolvedVia = $null - if ($src.Sid -and $bySid.ContainsKey($src.Sid)) { - $znUser = $bySid[$src.Sid]; $resolvedVia = 'SID' - } elseif ($src.Name -and $byPrincipal.ContainsKey($src.Name.ToLowerInvariant())) { - $znUser = $byPrincipal[$src.Name.ToLowerInvariant()]; $resolvedVia = 'PrincipalName' - } - - if (-not $znUser) { - # Typically well-known/system/machine accounts (SYSTEM, LOCAL SERVICE, *$) - no AD groups. - $unresolved.Add($userName) - Write-Verbose "Unresolved source user '$userName' (sid='$($src.Sid)') - skipped." - continue - } - # Case-variant userNames / multiple SIDs can map to one ZN user; count once. - if (-not $processedIds.Add($znUser.Id)) { - Write-Debug "Duplicate -> ZN user $($znUser.Id) already processed; skipping '$userName'." - continue +$work = @($sourceUsers.Values) +Write-Host "`nResolving $($work.Count) source user(s) and retrieving group memberships..." -ForegroundColor Cyan + +# Sequential resolver+fetcher (uses raw REST so behaviour matches the parallel path). +function Get-UserGroupsForSource { + param($Src) + $name = if ($Src.Name) { $Src.Name } else { $Src.Sid } + $uid = $null + try { + if ($Src.Name) { + $enc = [uri]::EscapeDataString($Src.Name) + $uid = (Invoke-WithRetry -Description "resolve $name" -Action { + Invoke-RestMethod -Uri "$baseUrl/users/searchIdByPrincipalName?principalName=$enc" -Headers $headers -Verbose:$false }).userId + } elseif ($Src.Sid) { + $enc = [uri]::EscapeDataString($Src.Sid) + $uid = (Invoke-WithRetry -Description "resolve $name" -Action { + Invoke-RestMethod -Uri "$baseUrl/users/searchIdBySid?sid=$enc" -Headers $headers -Verbose:$false }).userId + } + } catch { + $st = $null; try { $st = [int]$_.Exception.Response.StatusCode } catch {} + if ($st -ne 404) { Write-Warning "Resolve failed for '$name': $_" } } - $resolvedUsers.Add([pscustomobject]@{ Id = $znUser.Id; Name = $znUser.Name }) - Write-Debug "Resolved '$userName' via $resolvedVia -> ZN user $($znUser.Id)" + if (-not $uid) { return [pscustomobject]@{ Name = $name; UserId = $null; Groups = @() } } + + $groups = @() + try { + $groups = @((Invoke-WithRetry -Description "ancestors $uid" -Action { + Invoke-RestMethod -Uri "$baseUrl/users/$uid/ancestors" -Headers $headers -Verbose:$false }).items) + } catch { Write-Warning "Group lookup failed for '$name' ($uid): $_" } + [pscustomobject]@{ Name = $name; UserId = $uid; Groups = $groups } } -Write-Host " Source users resolved : $($resolvedUsers.Count) / $($sourceUsers.Count)" -ForegroundColor DarkGray -if ($unresolved.Count -gt 0) { - Write-Host " Unresolved (skipped) : $($unresolved.Count) (e.g. $((($unresolved | Select-Object -First 3) -join ', ')))" -ForegroundColor DarkGray -} - -# --------------------------------------------------------------------------- -# 4b. Retrieve each resolved user's group memberships. -# Parallel on PowerShell 7+ (capped by -MaxParallel); sequential on 5.1. -# Each call is retry-wrapped. Returns @{ Name; Groups }. -# --------------------------------------------------------------------------- -Write-Host "Retrieving group memberships for $($resolvedUsers.Count) user(s)..." -ForegroundColor Cyan - -# Parallelism only pays off once there are enough users to amortize runspace -# startup cost; below the threshold, sequential is faster. Tune via -MaxParallel. $parallelThreshold = 20 -$useParallel = ($PSVersionTable.PSVersion.Major -ge 7) -and ($MaxParallel -gt 1) -and ($resolvedUsers.Count -ge $parallelThreshold) -Write-Verbose ("Membership lookup mode: {0} ({1} users, PS {2})." -f ` +$useParallel = ($PSVersionTable.PSVersion.Major -ge 7) -and ($MaxParallel -gt 1) -and ($work.Count -ge $parallelThreshold) +Write-Verbose ("Lookup mode: {0} ({1} source users, PS {2})." -f ` $(if ($useParallel) {"parallel, throttle $MaxParallel"} else {"sequential (< $parallelThreshold users or PS<7 or -MaxParallel 1)"}), ` - $resolvedUsers.Count, $PSVersionTable.PSVersion) + $work.Count, $PSVersionTable.PSVersion) + +# Progress + ETA, updated as each user completes (works for both paths). +$progressActivity = "Resolving users + groups" +$swPhase = [System.Diagnostics.Stopwatch]::StartNew() +function Show-LookupProgress { + param([int]$Done, [int]$Total, [System.Diagnostics.Stopwatch]$Sw) + if ($Total -le 0) { return } + $status = "$Done / $Total users" + if ($Done -gt 0 -and $Done -lt $Total) { + $etaSec = [int](($Sw.Elapsed.TotalSeconds / $Done) * ($Total - $Done)) + $rate = [math]::Round($Done / [math]::Max($Sw.Elapsed.TotalSeconds, 0.001), 1) + $status += " (~{0} left, {1}/s)" -f ([TimeSpan]::FromSeconds($etaSec).ToString('mm\:ss')), $rate + } + Write-Progress -Activity $progressActivity -Status $status -PercentComplete ([int](($Done / $Total) * 100)) +} if ($useParallel) { - # Call the REST endpoint that Get-ZNUserMemberOf wraps (GET /users/{id}/ancestors) - # directly, so parallel runspaces do NOT each pay the (heavy) Import-Module cost. + # Raw REST in each runspace so they don't each pay the heavy Import-Module cost. $apiKey = $env:ZNApiKey $base = $baseUrl - $membershipResults = $resolvedUsers | ForEach-Object -ThrottleLimit $MaxParallel -Parallel { - $u = $_ + $collected = [System.Collections.Generic.List[object]]::new() + $work | ForEach-Object -ThrottleLimit $MaxParallel -Parallel { + $src = $_ $hdr = @{ Authorization = $using:apiKey } - $uri = "$using:base/users/$($u.Id)/ancestors" - $groups = $null - for ($attempt = 1; $attempt -le 4; $attempt++) { - try { - $groups = (Invoke-RestMethod -Uri $uri -Headers $hdr -Verbose:$false).items - break - } catch { - $status = $null; try { $status = [int]$_.Exception.Response.StatusCode } catch {} - $transient = ($status -in 429,500,502,503,504) -or ($null -eq $status) - if (-not $transient -or $attempt -eq 4) { - Write-Warning " Could not retrieve groups for $($u.Name) ($($u.Id)): $_" - break + $base = $using:base + $name = if ($src.Name) { $src.Name } else { $src.Sid } + + function Invoke-Req($uri) { + for ($a = 1; $a -le 4; $a++) { + try { return Invoke-RestMethod -Uri $uri -Headers $hdr -Verbose:$false } + catch { + $st = $null; try { $st = [int]$_.Exception.Response.StatusCode } catch {} + if ($st -eq 404) { throw } + $transient = ($st -in 429,500,502,503,504) -or ($null -eq $st) + if (-not $transient -or $a -eq 4) { throw } + Start-Sleep -Milliseconds ([int](1000 * [math]::Pow(2, $a-1) + (Get-Random -Maximum 250))) } - Start-Sleep -Milliseconds ([int](1000 * [math]::Pow(2, $attempt-1) + (Get-Random -Maximum 250))) } } - [pscustomobject]@{ Name = $u.Name; Groups = @($groups) } - } -} else { - $membershipResults = foreach ($u in $resolvedUsers) { - $groups = $null + + $uid = $null try { - $groups = Invoke-WithRetry -Description "Get-ZNUserMemberOf $($u.Id)" -Action { - (Get-ZNUserMemberOf -UserId $u.Id -Verbose:$false -Debug:$false).Items + if ($src.Name) { + $enc = [uri]::EscapeDataString($src.Name) + $uid = (Invoke-Req "$base/users/searchIdByPrincipalName?principalName=$enc").userId + } elseif ($src.Sid) { + $enc = [uri]::EscapeDataString($src.Sid) + $uid = (Invoke-Req "$base/users/searchIdBySid?sid=$enc").userId } } catch { - Write-Warning " Could not retrieve groups for $($u.Name) ($($u.Id)): $_" + $st = $null; try { $st = [int]$_.Exception.Response.StatusCode } catch {} + if ($st -ne 404) { Write-Warning "Resolve failed for '$name': $_" } } - Write-Verbose "User '$($u.Name)' ($($u.Id)) is a member of $(@($groups).Count) group(s)." - [pscustomobject]@{ Name = $u.Name; Groups = @($groups) } + if (-not $uid) { + [pscustomobject]@{ Name = $name; UserId = $null; Groups = @() } + } else { + $g = @() + try { $g = @((Invoke-Req "$base/users/$uid/ancestors").items) } + catch { Write-Warning "Group lookup failed for '$name' ($uid): $_" } + [pscustomobject]@{ Name = $name; UserId = $uid; Groups = $g } + } + } | ForEach-Object { + # Runs on the main thread as each parallel result streams back. + $collected.Add($_) + Show-LookupProgress -Done $collected.Count -Total $work.Count -Sw $swPhase + } + $rawResults = $collected +} else { + $done = 0 + $rawResults = foreach ($src in $work) { + $done++ + Show-LookupProgress -Done $done -Total $work.Count -Sw $swPhase + Get-UserGroupsForSource -Src $src } } +Write-Progress -Activity $progressActivity -Completed +Write-Verbose ("Lookup phase completed in {0:n1}s." -f $swPhase.Elapsed.TotalSeconds) # --------------------------------------------------------------------------- -# 4c. Merge memberships into the group -> users map (sequential; no races). +# 4b. Merge results into the group -> users map (sequential; dedupe by user id). # --------------------------------------------------------------------------- -# groupId -> @{ Name; Id; Users (list of display names) } -$groupUserMap = [ordered]@{} -foreach ($res in $membershipResults) { - foreach ($g in $res.Groups) { +$groupUserMap = [ordered]@{} # groupId -> @{ Name; Id; Users } +$processedIds = [System.Collections.Generic.HashSet[string]]::new() +$unresolved = [System.Collections.Generic.List[string]]::new() +$resolvedCount = 0 + +foreach ($r in $rawResults) { + if (-not $r) { continue } + if (-not $r.UserId) { + $unresolved.Add($r.Name) + Write-Verbose "Unresolved source user '$($r.Name)' - skipped (not a ZN user)." + continue + } + if (-not $processedIds.Add($r.UserId)) { + Write-Debug "Duplicate -> ZN user $($r.UserId) ('$($r.Name)') already counted; skipping." + continue + } + $resolvedCount++ + Write-Verbose "User '$($r.Name)' ($($r.UserId)) is a member of $(@($r.Groups).Count) group(s)." + + foreach ($g in $r.Groups) { if (-not $g) { continue } - $gid = $g.Id + $gid = $g.id if (-not $groupUserMap.Contains($gid)) { $groupUserMap[$gid] = @{ - Name = $g.Name + Name = $g.name Id = $gid Users = [System.Collections.Generic.List[string]]::new() } } - $groupUserMap[$gid].Users.Add($res.Name) + $groupUserMap[$gid].Users.Add($r.Name) } } +Write-Host " Source users resolved : $resolvedCount / $($work.Count)" -ForegroundColor DarkGray +if ($unresolved.Count -gt 0) { + Write-Host " Unresolved (skipped) : $($unresolved.Count) (e.g. $((($unresolved | Select-Object -First 3) -join ', ')))" -ForegroundColor DarkGray +} + # --------------------------------------------------------------------------- # 5. Filter groups by threshold and build report # --------------------------------------------------------------------------- diff --git a/Utils/README.md b/Utils/README.md index 2c0094a..5f5388c 100644 --- a/Utils/README.md +++ b/Utils/README.md @@ -44,7 +44,7 @@ If `$env:ZNApiKey` is not set the script exits with an authentication error. | `-FromDays` | no | `30` | How many days of activity history to scan (`1`–`365`). | | `-OutputPath` | no | `./CommonGroups__.csv` | Output CSV path. | | `-IncludeSourceUsers` | no | off | Switch. Adds a `SourceUsers` column to the CSV listing the member display names per group. Off by default — groups containing thousands of users make the column very noisy. `SourceUserCount` is always present. | -| `-MaxParallel` | no | `8` | Max concurrent group-membership lookups (`1`–`16`). Used only on PowerShell 7+ and only once there are at least 20 resolved users to amortize runspace startup. Set to `1` to force sequential, lower if the tenant returns HTTP 429. Ignored on Windows PowerShell 5.1. | +| `-MaxParallel` | no | `8` | Max concurrent per-user resolve + group-membership lookups (`1`–`64`). Used only on PowerShell 7+ and only once there are at least 20 source users to amortize runspace startup. Raise (e.g. `24`–`32`) for very large destinations with thousands of source users; set `1` to force sequential, or lower if the tenant returns HTTP 429. Ignored on Windows PowerShell 5.1. | `-DestinationFQDN` and `-DestinationIP` are mutually exclusive — supply exactly one. Running the script with neither prints usage and exits. @@ -57,13 +57,17 @@ If `$env:ZNApiKey` is not set the script exits with an authentication error. 3. **Collects distinct source users** in one of two ways: - **Fast path (primary):** calls `/activities/network/distinctField/srcUser`, which returns the de-duplicated set of source users (with per-user hit counts) in a single request. This avoids paging through tens of thousands of activity rows on busy destinations. - **Fallback:** if the `distinctField` endpoint is unavailable (older tenant), the script pages `/activities/network` and de-duplicates client-side. -4. Pages through `Get-ZNUser` to build a SID → user and PrincipalName → user lookup, so activity SIDs that don't match ZN user IDs (e.g. Entra synthetic SIDs) can still be resolved. -5. **Resolves** each distinct source user against that lookup (SID first, then PrincipalName). If two source entries map to the same ZN user (case-variant `userName`, or multiple SIDs for one identity) the user is counted **once**. Unresolvable system/machine accounts are skipped and reported. -6. **Retrieves group memberships** for each resolved user: - - On **PowerShell 7+** with ≥ 20 resolved users, the script fans out to `-MaxParallel` concurrent runspaces (default 8) calling `/users/{id}/ancestors` directly — this avoids the per-runspace cost of re-importing the ZN module. - - On **Windows PowerShell 5.1**, on PS 7+ with fewer than 20 resolved users, or when `-MaxParallel 1` is supplied, the script runs sequentially via `Get-ZNUserMemberOf`. - - All API calls (distinctField, activity pages, `Get-ZNUser`, membership lookups) are wrapped with retry + exponential backoff + jitter on transient failures (HTTP 429 / 5xx and transport errors). `Retry-After` headers are honoured. -7. Merges memberships into a group → users map (sequential, race-free), filters out groups below `-MinUserCount`, sorts by member count desc, writes the CSV, and prints the top 10 to the console. +4. **Resolves + retrieves groups per user, on demand.** Rather than enumerating the whole ZN directory (which can be 100k+ users on a large tenant), each source user is looked up individually: + - `GET /users/searchIdByPrincipalName?principalName=DOMAIN\user` (fast path's `userName`) + - `GET /users/searchIdBySid?sid=` (used by the activity-scan fallback when only a SID is available) + - `HTTP 404` is treated as "not a ZN user" (system/machine/unknown account) and skipped silently — those users won't have AD groups anyway. + - Once resolved, `GET /users/{id}/ancestors` returns the user's groups in one call. +5. **Parallel vs sequential:** + - On **PowerShell 7+** with ≥ 20 source users, resolve-and-fetch runs in `-MaxParallel` concurrent runspaces (default 8). Each runspace uses raw REST — no per-runspace module import. Results stream back to the main thread with live progress + ETA. + - On **Windows PowerShell 5.1**, on PS 7+ with fewer than 20 source users, or when `-MaxParallel 1` is supplied, the script runs sequentially. + - All API calls (distinctField, activity pages, resolve, ancestors) are wrapped with retry + exponential backoff + jitter on transient failures (HTTP 429 / 5xx and transport errors). `Retry-After` headers are honoured. `HTTP 404` from the resolver short-circuits without retrying. +6. **Merges** the per-user results into a group → users map (sequential, race-free) and **dedupes by ZN user id**, so case-variant `userName`s or multiple SIDs that resolve to the same ZN user are counted **once**. Unresolvable users are reported as a skipped count. +7. Filters out groups below `-MinUserCount`, sorts by member count desc, writes the CSV, and prints the top 10 to the console. ### Output @@ -113,7 +117,7 @@ Get-Help .\Find-CommonGroupsForDestination.ps1 -Full ### Notes & caveats -- The fast path (`distinctField/srcUser`) returns only `userName` (`DOMAIN\user`), so users are resolved by `PrincipalName`. The fallback activity-scan path also captures the SID and tries SID first. Either way, well-known/system/machine accounts (`SYSTEM`, `LOCAL SERVICE`, `*$`, etc.) won't resolve to a ZN user and are skipped — the count of unresolved users is reported. +- The fast path (`distinctField/srcUser`) returns only `userName` (`DOMAIN\user`), so users are resolved via `searchIdByPrincipalName`. The fallback activity-scan path also captures the SID, in which case `searchIdBySid` is used. Well-known/system/machine accounts (`SYSTEM`, `LOCAL SERVICE`, `*$`, etc.) return `HTTP 404` and are skipped — the count of unresolved users is reported. - Only activities that include source-user information are considered. Service-to-service or agentless flows without a resolvable user contribute nothing to the report. - If two source entries map to the same ZN user, the user is counted **once** per group (no double-counting from SID/name variants). - The activity-feed fallback path uses cursor pagination at `_limit=400`; very long lookbacks on busy destinations will take longer and consume more API calls. The fast path is unaffected.