Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/coverage.yml
Original file line number Diff line number Diff line change
Expand Up @@ -196,7 +196,7 @@ jobs:
shell: pwsh
continue-on-error: true
run: |
Install-Module PSComplexity -RequiredVersion 0.1.0 -Force -Scope CurrentUser
Install-Module PSComplexity -RequiredVersion 0.3.0 -Force -Scope CurrentUser
$psDir = 'docs/coverage/powershell'
New-Item -ItemType Directory -Path $psDir -Force | Out-Null
& tools/complexity/measure_ps.ps1 | Set-Content (Join-Path $psDir 'complexity.json') -Encoding utf8
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/pr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -874,7 +874,7 @@ jobs:
# The PowerShell measurer is the published PSComplexity module.
- name: Install PSComplexity (PowerShell complexity measurer)
shell: pwsh
run: Install-Module PSComplexity -RequiredVersion 0.1.0 -Force -Scope CurrentUser
run: Install-Module PSComplexity -RequiredVersion 0.3.0 -Force -Scope CurrentUser

# Validate the ratchet gate logic (Python) before it gates anything.
- name: Test the ratchet gate logic
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/ps-mutation.yml
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ jobs:
shell: pwsh
run: |
Install-Module Pester -RequiredVersion 5.8.0 -Force -Scope CurrentUser
Install-Module PSMutant -RequiredVersion 0.2.0 -Force -Scope CurrentUser
Install-Module PSMutant -RequiredVersion 0.3.2 -Force -Scope CurrentUser

- name: Run mutation testing (enforced — fails below the break floor)
shell: pwsh
Expand Down
2 changes: 2 additions & 0 deletions changes/bump-ps-quality-tools.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
- Upgraded the PowerShell complexity measurer to PSComplexity 0.3.0, which now scores the branching PowerShell expresses through its own flow constructs — `ForEach-Object`/`Where-Object` (and aliases), the `&&`/`||` pipeline chains, and the `??`/`??=` operators — that earlier versions read as straight-line code, and no longer reports a passing gate when it silently measured nothing.
- Upgraded the PowerShell mutation-testing tool to PSMutant 0.3.2, which refuses a per-mutant timeout too small to run the real suite (instead of reporting a flattering perfect score) and refuses a config path that escapes the source root.
2 changes: 1 addition & 1 deletion docs/contributing/writing-tests-that-assert.md
Original file line number Diff line number Diff line change
Expand Up @@ -145,7 +145,7 @@ Config: `.ci/psmutant.config.json`. Every eligible crawler file must be in
`test/unit/PSMutationScope.Tests.ps1`.

```bash
Install-Module PSMutant -RequiredVersion 0.2.0
Install-Module PSMutant -RequiredVersion 0.3.2
Invoke-PSMutation -ConfigFile .ci/psmutant.config.json -SourceRoot .
```

Expand Down
3 changes: 1 addition & 2 deletions test/Run-AllTests.ps1
Original file line number Diff line number Diff line change
Expand Up @@ -128,8 +128,7 @@ function Show-Summary {
Write-Host "╠══════════════════════════════════════════════════╣" -ForegroundColor Cyan

foreach ($r in $script:PhaseResults) {
$icon = if ($r.Passed) { "✓" } else { "✗" }
$color = if ($r.Passed) { "Green" } else { "Red" }
$icon, $color = if ($r.Passed) { "✓", "Green" } else { "✗", "Red" }
$line = " $icon $($r.Phase)".PadRight(42) + "$($r.Duration)s"
Write-Host "║ $line ║" -ForegroundColor $color
}
Expand Down
20 changes: 14 additions & 6 deletions test/unit/Test-GraphAPI.ps1
Original file line number Diff line number Diff line change
Expand Up @@ -353,15 +353,23 @@ function Invoke-TokenManagementTest {
}

# Per-category pass/skip breakdown for the summary.
function Get-TestCategoryCounts {
param($Group)
return @{
Passed = ($Group | Where-Object { $_.Passed -and -not $_.Skipped }).Count
Skipped = ($Group | Where-Object Skipped).Count
Failed = ($Group | Where-Object { -not $_.Passed -and -not $_.Skipped }).Count
Total = $Group.Count
}
}

function Write-CategoryBreakdown {
$categories = $script:TestResults | Group-Object Category
foreach ($cat in $categories) {
$passed = ($cat.Group | Where-Object { $_.Passed -and -not $_.Skipped }).Count
$skipped = ($cat.Group | Where-Object Skipped).Count
$total = $cat.Group.Count
$color = if (($cat.Group | Where-Object { -not $_.Passed -and -not $_.Skipped }).Count -eq 0) { "Green" } else { "Yellow" }
$skipText = if ($skipped -gt 0) { " ($skipped skipped)" } else { "" }
Write-Host " $($cat.Name): $passed/$total$skipText" -ForegroundColor $color
$c = Get-TestCategoryCounts -Group $cat.Group
$color = if ($c.Failed -eq 0) { "Green" } else { "Yellow" }
$skipText = if ($c.Skipped -gt 0) { " ($($c.Skipped) skipped)" } else { "" }
Write-Host " $($cat.Name): $($c.Passed)/$($c.Total)$skipText" -ForegroundColor $color
}
}

Expand Down
12 changes: 7 additions & 5 deletions tools/complexity/measure_ps.ps1
Original file line number Diff line number Diff line change
Expand Up @@ -14,9 +14,11 @@
[ { "file": "<repo-relative>", "unit": "<name|<script-body>>", "line": <int>,
"cc": <int>, "cog": <int> }, ... ]

Cyclomatic numbers are identical to the previous bundled measurer; cognitive matches
except where PSComplexity is more faithful (it also counts recursion and labelled
break/continue). The baselines under .ci/ are generated from this output.
As of PSComplexity 0.3.0 both metrics also score the branching PowerShell expresses
through its own flow constructs -- ForEach-Object / Where-Object (and aliases), the
&& / || pipeline chains, and the ?? / ??= operators -- which earlier versions read as
straight-line code. A pipeline body now costs exactly what the equivalent keyword form
costs. The baselines under .ci/ are generated from this output.

.OUTPUTS
JSON array to stdout.
Expand All @@ -29,8 +31,8 @@ param(

$ErrorActionPreference = 'Stop'

if (-not (Get-Module PSComplexity -ListAvailable | Where-Object Version -ge '0.1.0')) {
Install-Module PSComplexity -RequiredVersion 0.1.0 -Force -Scope CurrentUser
if (-not (Get-Module PSComplexity -ListAvailable | Where-Object Version -ge '0.3.0')) {
Install-Module PSComplexity -RequiredVersion 0.3.0 -Force -Scope CurrentUser
}
Import-Module PSComplexity

Expand Down
46 changes: 46 additions & 0 deletions tools/crawlers/entra-id/EntraIDCrawler.Functions.ps1
Original file line number Diff line number Diff line change
Expand Up @@ -536,3 +536,49 @@ function Format-FGDelegatedPermissionName {
if ($ClientName) { "$Scope on $TargetName (via $ClientName)" }
else { "$Scope on $TargetName" }
}

# Split a Graph /delta response into the live records and the @removed tombstone
# ids. Shared by the users and service-principal delta fetches, which did this
# identical split inline. Returns @{ items; removedIds }.
function Split-FGDeltaResponse {
[CmdletBinding()]
param($Response)
$items = @($Response.value | Where-Object { -not $_.'@removed' })
$removed = @($Response.value | Where-Object { $_.'@removed' } | ForEach-Object { $_.id })
return @{ items = $items; removedIds = $removed }
}

# Fold one PIM eligibility batch into $RecordsList (by reference) and return the
# count of distinct source groups the batch touched. Extracted from Sync-EntraPim's
# per-batch loop so the phase stays under the complexity ceiling.
function Add-EntraPimBatchRecords {
[CmdletBinding()]
param($BatchOutput, $RecordsList)
$groupSet = @{}
foreach ($r in $BatchOutput) {
$RecordsList.Add((ConvertTo-EntraPimRecord -EligibilityRow $r))
$groupSet[$r.resourceId] = $true
}
return $groupSet.Count
}

# Stream one sign-in-log day slice into $Aggregate (by reference), folding each
# event via Add-EntraSignInEventToAggregate. Returns @{ count; skipped }. The
# counters live in a hashtable so the increments survive the streaming pipeline
# block (a plain local wouldn't propagate out of ForEach-Object).
function Invoke-EntraSignInSlice {
[CmdletBinding()]
param(
[Parameter(Mandatory)] [string]$SliceUri,
[Parameter(Mandatory)] [hashtable]$Aggregate,
[Parameter(Mandatory)] [hashtable]$AppIdToSpId
)
$counters = @{ count = 0; skipped = 0 }
Invoke-FGGetRequestStream -URI $SliceUri | ForEach-Object {
if (-not (Add-EntraSignInEventToAggregate -SignInEvent $_ -Aggregate $Aggregate -AppIdToSpId $AppIdToSpId)) {
$counters.skipped++
}
$counters.count++
}
return $counters
}
54 changes: 19 additions & 35 deletions tools/crawlers/entra-id/EntraIDCrawler.Phases.ps1
Original file line number Diff line number Diff line change
Expand Up @@ -719,13 +719,8 @@ function Sync-EntraPim {

$batchOutput = Invoke-FGGroupPimBatchParallel -Batch @($batch) -Token $token -ThrottleLimit 16

# Group output by source group to compute pimGroupCount accurately
$groupSet = @{}
foreach ($r in $batchOutput) {
$pimRecordsList.Add((ConvertTo-EntraPimRecord -EligibilityRow $r))
$groupSet[$r.resourceId] = $true
}
$pimGroupCount += $groupSet.Count
# Fold the batch into $pimRecordsList and count the distinct groups it touched.
$pimGroupCount += Add-EntraPimBatchRecords -BatchOutput $batchOutput -RecordsList $pimRecordsList

$pimChecked = [Math]::Min($i + $pimBatchSize, $pimTotal)
$subPct = 61 + [int](([double]$pimChecked / $pimTotal) * 4)
Expand All @@ -736,12 +731,9 @@ function Sync-EntraPim {
Write-Host " Found $pimGroupCount PIM-enabled group(s) with $($pimRecords.Count) eligible memberships" -ForegroundColor Gray

if ($pimRecords.Count -gt 0) {
# Dedup by (resourceId, principalId)
$seen = @{}
$pimRecords = @($pimRecords | Where-Object {
$k = "$($_.resourceId)|$($_.principalId)"
if ($seen.ContainsKey($k)) { $false } else { $seen[$k] = $true; $true }
})
# Dedup by (resourceId, principalId) — HashSet.Add returns $false for a dup.
$seen = [System.Collections.Generic.HashSet[string]]::new()
$pimRecords = @($pimRecords | Where-Object { $seen.Add("$($_.resourceId)|$($_.principalId)") })
Send-IngestBatch -Endpoint 'ingest/resource-assignments' -SystemId $SystemId -SyncMode 'full' `
-Scope @{ assignmentType = 'Eligible'; resourceType = 'Group' } -Records $pimRecords
}
Expand Down Expand Up @@ -793,8 +785,9 @@ function Get-EntraServicePrincipalData {
try {
$deltaUri = "https://graph.microsoft.com/beta/servicePrincipals/delta?`$deltatoken=$([uri]::EscapeDataString($spsToken))"
$resp = Invoke-FGGetDeltaRequest -URI $deltaUri
$sps = @($resp.value | Where-Object { -not $_.'@removed' })
$removedSpIds = @($resp.value | Where-Object { $_.'@removed' } | ForEach-Object { $_.id })
$split = Split-FGDeltaResponse -Response $resp
$sps = $split.items
$removedSpIds = $split.removedIds
$newSpsToken = $resp.deltaToken
$spDeltaHit = $true
Write-Host " Delta: $($sps.Count) changed + $($removedSpIds.Count) removed" -ForegroundColor Gray
Expand Down Expand Up @@ -1024,11 +1017,10 @@ function Sync-EntraSignInLogs {
$totalEvents = 0
$sliceFailures = @()

# Per-event aggregation lives in Add-EntraSignInEventToAggregate
# (EntraIDCrawler.Transform.ps1): it folds one event into $agg (by
# reference) and returns $false when skipped. The skip counter stays in
# $script: scope so it survives the streaming ForEach-Object below (a
# plain local wouldn't propagate out of the pipeline block).
# Per-slice streaming lives in Invoke-EntraSignInSlice
# (EntraIDCrawler.Functions.ps1): it folds each event into $agg (by
# reference) via Add-EntraSignInEventToAggregate and returns the event and
# skip counts for the slice, which we accumulate here.
$script:_signin_skipped = 0

$nowUtc = (Get-Date).ToUniversalTime()
Expand All @@ -1038,20 +1030,11 @@ function Sync-EntraSignInLogs {
$sliceFilter = [uri]::EscapeDataString("createdDateTime ge $sliceStart and createdDateTime lt $sliceEnd")
$sliceUri = "https://graph.microsoft.com/beta/auditLogs/signIns?`$filter=$sliceFilter&`$top=999"
Update-CrawlerProgress -Detail "Fetching day slice $($d + 1)/${SignInLogsDays}: $sliceStart..$sliceEnd"
$sliceCount = 0
try {
# IMPORTANT: pipe directly into ForEach-Object so each Graph
# page can be GC'd as soon as it's aggregated. Assigning the
# result to a variable first would buffer the whole slice
# and defeat the streaming.
Invoke-FGGetRequestStream -URI $sliceUri | ForEach-Object {
if (-not (Add-EntraSignInEventToAggregate -SignInEvent $_ -Aggregate $agg -AppIdToSpId $appIdToSpId)) {
$script:_signin_skipped++
}
$sliceCount++
}
$totalEvents += $sliceCount
Write-Host " Slice $($d + 1)/$SignInLogsDays ($sliceStart..$sliceEnd): $sliceCount events" -ForegroundColor Gray
$sliceResult = Invoke-EntraSignInSlice -SliceUri $sliceUri -Aggregate $agg -AppIdToSpId $appIdToSpId
$script:_signin_skipped += $sliceResult.skipped
$totalEvents += $sliceResult.count
Write-Host " Slice $($d + 1)/$SignInLogsDays ($sliceStart..$sliceEnd): $($sliceResult.count) events" -ForegroundColor Gray
} catch {
# One bad slice (typically an expired skiptoken 400 deep in
# pagination) doesn't abort the whole phase — we record it
Expand Down Expand Up @@ -1462,8 +1445,9 @@ function Get-EntraUserData {
try {
$deltaUri = "https://graph.microsoft.com/beta/users/delta?`$deltatoken=$([uri]::EscapeDataString($usersToken))"
$resp = Invoke-FGGetDeltaRequest -URI $deltaUri
$users = @($resp.value | Where-Object { -not $_.'@removed' })
$removedUserIds = @($resp.value | Where-Object { $_.'@removed' } | ForEach-Object { $_.id })
$split = Split-FGDeltaResponse -Response $resp
$users = $split.items
$removedUserIds = $split.removedIds
$newUsersToken = $resp.deltaToken
$deltaHit = $true
Write-Host " Delta: $($users.Count) changed + $($removedUserIds.Count) removed" -ForegroundColor Gray
Expand Down
25 changes: 19 additions & 6 deletions tools/crawlers/midpoint/MidpointCrawler.Transform.ps1
Original file line number Diff line number Diff line change
Expand Up @@ -103,6 +103,22 @@ function ConvertTo-MidpointOrgContextRecord {
# Topologically sorts context records so a parent precedes its children. A parent
# OID outside the synced set is treated as a root (its parentContextId is nulled
# out to avoid an FK violation). Verbatim from the inline Orgs-phase sort.
# Place one record into $Sorted if it is ready (a root, or its parent is already
# inserted, or its parent is outside the synced set — in which case it is nulled to
# a root). Returns $true when placed, $false when it must wait another pass.
function Add-MidpointReadyContext {
[CmdletBinding()]
param($Rec, $Present, $Inserted, $Sorted)
$p = $Rec.parentContextId
if (-not $p -or -not $Present.Contains($p) -or $Inserted.Contains($p)) {
# A parent outside the synced set is treated as a root (null it out)
if ($p -and -not $Present.Contains($p)) { $Rec.parentContextId = $null }
$Sorted.Add($Rec); [void]$Inserted.Add($Rec.id)
return $true
}
return $false
}

function Get-MidpointContextsInTopologicalOrder {
[CmdletBinding()]
param($Records)
Expand All @@ -115,12 +131,9 @@ function Get-MidpointContextsInTopologicalOrder {
while ($remaining.Count -gt 0 -and $pass -lt $maxPass) {
$pass++; $next = [System.Collections.Generic.List[object]]::new()
foreach ($rec in $remaining) {
$p = $rec.parentContextId
# A parent outside the synced set is treated as a root (null it out)
if (-not $p -or -not $present.Contains($p) -or $inserted.Contains($p)) {
if ($p -and -not $present.Contains($p)) { $rec.parentContextId = $null }
$sorted.Add($rec); [void]$inserted.Add($rec.id)
} else { $next.Add($rec) }
if (-not (Add-MidpointReadyContext -Rec $rec -Present $present -Inserted $inserted -Sorted $sorted)) {
$next.Add($rec)
}
}
$remaining = $next
}
Expand Down
43 changes: 43 additions & 0 deletions tools/crawlers/omada/OmadaCrawler.Functions.ps1
Original file line number Diff line number Diff line change
Expand Up @@ -116,4 +116,47 @@ function Send-IngestBatch {
-Records $Records -DeletedIds $DeletedIds -BatchSize $BatchSize
}

# Shape the raw OData items for one context entity set into ingest-ready context
# records, dropping any without an externalId/displayName. Orgunit carries a parent
# hierarchy, so its records are topologically sorted (parent before child).
# Extracted from Sync-OmadaContexts to keep that phase under the complexity ceiling.
function Build-OmadaContextRecords {
[CmdletBinding()]
param($Items, [string]$EntitySet, [string]$ContextType)
if ($EntitySet -eq 'Orgunit') {
$RawRecords = @($Items | ForEach-Object {
ConvertTo-OmadaOrgUnitContextRecord -OrgUnit $_ -DefaultContextType $ContextType
} | Where-Object { $_.externalId -and $_.displayName })
return @(Get-OmadaContextsInTopologicalOrder -Records $RawRecords)
}
return @($Items | ForEach-Object {
ConvertTo-OmadaFlatContextRecord -Item $_ -ContextType $ContextType
} | Where-Object { $_.externalId -and $_.displayName })
}

# Combine the role and CRA assignments for one Omada system, dedup by
# (principalId, resourceId), and ingest them as governed Direct assignments.
# Returns the inserted count. Extracted from Send-OmadaGovernanceAssignments's
# per-system loop so that phase stays under the complexity ceiling.
function Send-OmadaGovernanceAssignmentForSystem {
[CmdletBinding()]
param(
[string]$Key,
[hashtable]$RaBySys = @{},
[hashtable]$AssignmentsBySys = @{},
[hashtable]$OmadaSystemMap = @{},
[int]$SystemId
)
$SysId = if ($Key -eq '__main__') { $SystemId } else { $OmadaSystemMap[$Key] }
$Combined = [System.Collections.Generic.List[object]]::new()
if ($RaBySys.ContainsKey($Key)) { $Combined.AddRange($RaBySys[$Key]) }
if ($AssignmentsBySys.ContainsKey($Key)) { $Combined.AddRange($AssignmentsBySys[$Key]) }
$Seen = [System.Collections.Generic.HashSet[string]]::new()
$Dedup = @($Combined | Where-Object { $Seen.Add("$($_.principalId)|$($_.resourceId)") })
if ($Dedup.Count -eq 0) { return 0 }
$R = Send-IngestBatch -Endpoint 'ingest/resource-assignments' -SystemId $SysId `
-SyncMode 'full' -Scope @{ assignmentType = 'Direct'; governed = $true } -Records $Dedup
return ($R.inserted ?? 0)
}

#endregion Functions
Loading
Loading