Improve diagnostic logging for customer issue analysis

[JayVDZ]

Summary

When investigating sync issues (particularly reconciliation failures), the current logging doesn't provide sufficient detail to diagnose why specific attribute changes weren't confirmed. This makes it difficult for both developers and customers to understand and resolve issues.

Background

During investigation of an intermittent DN construction issue, we found that:

1. When attributes fail to confirm during reconciliation, we log that retry is needed but not why (expected vs actual values)
2. There's no easy way to correlate logs for a specific object's journey through the sync process
3. The distinction between Debug and Verbose logging levels isn't being fully utilised

Current State

• Serilog with configurable minimum levels via JIM_LOG_LEVEL environment variable
• File logging with daily rolling and 31-day retention
• Console output for real-time viewing
• Structured logging with Log.Information(), Log.Debug(), etc.
• Performance diagnostics via Diagnostics.EnableLogging()

Recommendations

Short-term (Low Effort)

1. Expected vs Actual Logging - Add expected vs actual value logging in reconciliation at Debug level
2. Object ID Correlation - Add CSO ID, MVO ID, and PendingExport ID to key log messages
3. Verbose Level Usage - Use Log.Verbose() for very detailed tracing (attribute-by-attribute comparisons)

Medium-term

1. Scoped Logging Context - Push object IDs to LogContext so all nested logs include them automatically
2. Diagnostic Mode Toggle - Admin UI option to temporarily increase log level for a specific sync run
3. Log Export API - Endpoint to download recent logs filtered by date/severity

Long-term

1. Structured Log Aggregation - Integration with Seq or similar for searching structured log properties
2. Per-Object Trace Mode - Ability to enable verbose logging for specific objects
3. Activity-Based Log Grouping - Associate all logs for a sync activity together for easy extraction

Customer Use Case

When customers experience issues, we want to be able to:

1. Ask them to increase the logging level (e.g., to Debug or Verbose)
2. Have them reproduce the issue
3. Collect the logs and analyse them to identify the root cause

The short-term improvements would make this workflow much more effective.

Related Files

• JIM.Application/Services/PendingExportReconciliationService.cs - Reconciliation logging
• JIM.Worker/Worker.cs - Logging initialisation
• JIM.Web/Program.cs - Web logging configuration
• JIM.Application/Servers/ExportEvaluationServer.cs - Export evaluation logging