Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions src/NFury/Commands/Server/ServerCommand.cs
Original file line number Diff line number Diff line change
Expand Up @@ -252,6 +252,12 @@ public override async Task<int> ExecuteAsync(CommandContext context, ServerSetti
return Results.Ok(executions);
});

app.MapPost("/api/executions/compare", async (ExecutionComparisonRequest request, ExecutionService service) =>
{
Copy link

Copilot AI Jan 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comparison endpoint lacks input validation. The API should validate that baselineExecutionId and compareExecutionId are positive integers and that they are not equal to each other. While the UI prevents selecting the same execution twice, API endpoints should always validate inputs independently of the client.

Suggested change
{
{
if (request is null)
{
return Results.BadRequest(new ErrorResponse("Request body is required"));
}
if (request.BaselineExecutionId <= 0 || request.CompareExecutionId <= 0)
{
return Results.BadRequest(new ErrorResponse("BaselineExecutionId and CompareExecutionId must be positive integers"));
}
if (request.BaselineExecutionId == request.CompareExecutionId)
{
return Results.BadRequest(new ErrorResponse("BaselineExecutionId and CompareExecutionId must not be equal"));
}

Copilot uses AI. Check for mistakes.
var result = await service.CompareExecutionsAsync(request.BaselineExecutionId, request.CompareExecutionId);
return result != null ? Results.Ok(result) : Results.NotFound(new ErrorResponse("One or both executions not found"));
});

AnsiConsole.MarkupLine($"[bold green]NFury Web Server started![/]");
AnsiConsole.MarkupLine($"[blue]Open your browser at:[/] [link]http://{settings.Host}:{settings.Port}[/]");
AnsiConsole.MarkupLine($"[dim]Database:[/] {dbPath}");
Expand Down
4 changes: 2 additions & 2 deletions src/NFury/Commands/Server/ServerSettings.cs
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,8 @@ namespace NFury.Commands.Server;
public class ServerSettings : CommandSettings
{
[CommandOption("-p|--port")]
[DefaultValue(5000)]
[Description("Define the port for the web server. Default is 5000.")]
[DefaultValue(5002)]
[Description("Define the port for the web server. Default is 5002.")]
Comment on lines +9 to +10
Copy link

Copilot AI Jan 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The default port has been changed from 5000 to 5002 without explanation in the PR description. This appears to be an unintended change that is unrelated to the execution comparison feature. Changing the default port is a breaking change for existing users who may have configurations, scripts, or firewall rules based on port 5000. If this change is intentional, it should be documented in the PR description with justification.

Suggested change
[DefaultValue(5002)]
[Description("Define the port for the web server. Default is 5002.")]
[DefaultValue(5000)]
[Description("Define the port for the web server. Default is 5000.")]

Copilot uses AI. Check for mistakes.
public int Port { get; set; }

[CommandOption("--host")]
Expand Down
4 changes: 4 additions & 0 deletions src/NFury/Web/JsonContext.cs
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,10 @@ public record ExecutionListResponse(List<TestExecution> Executions, int Total);
[JsonSerializable(typeof(EndpointExportData))]
[JsonSerializable(typeof(ExecutionExportData))]
[JsonSerializable(typeof(ProjectImportResult))]
[JsonSerializable(typeof(ExecutionComparisonRequest))]
[JsonSerializable(typeof(ExecutionComparisonResult))]
[JsonSerializable(typeof(ExecutionSummary))]
[JsonSerializable(typeof(PerformanceDelta))]
[JsonSerializable(typeof(List<EndpointExportData>))]
[JsonSerializable(typeof(List<ExecutionExportData>))]
[JsonSerializable(typeof(List<Project>))]
Expand Down
204 changes: 204 additions & 0 deletions src/NFury/Web/Models.cs
Original file line number Diff line number Diff line change
Expand Up @@ -440,3 +440,207 @@ public record SignalRTestErrorMessage
/// </summary>
public string Error { get; init; } = string.Empty;
}

/// <summary>
/// Request to compare two executions
/// </summary>
public record ExecutionComparisonRequest
{
/// <summary>
/// The ID of the baseline execution
/// </summary>
public int BaselineExecutionId { get; init; }

/// <summary>
/// The ID of the execution to compare against the baseline
/// </summary>
public int CompareExecutionId { get; init; }
}

/// <summary>
/// Result of comparing two test executions
/// </summary>
public record ExecutionComparisonResult
{
/// <summary>
/// The baseline execution details
/// </summary>
public ExecutionSummary Baseline { get; init; } = new();

/// <summary>
/// The comparison execution details
/// </summary>
public ExecutionSummary Compare { get; init; } = new();

/// <summary>
/// Performance differences between the two executions
/// </summary>
public PerformanceDelta Delta { get; init; } = new();
}

/// <summary>
/// Summary of an execution for comparison purposes
/// </summary>
public record ExecutionSummary
{
/// <summary>
/// The execution ID
/// </summary>
public int Id { get; init; }

/// <summary>
/// The test identifier
/// </summary>
public string TestId { get; init; } = string.Empty;

/// <summary>
/// The endpoint name if linked to an endpoint
/// </summary>
public string? EndpointName { get; init; }

/// <summary>
/// The target URL tested
/// </summary>
public string Url { get; init; } = string.Empty;

/// <summary>
/// When the test started
/// </summary>
public DateTime StartedAt { get; init; }

/// <summary>
/// Total requests made
/// </summary>
public long TotalRequests { get; init; }

/// <summary>
/// Failed requests count
/// </summary>
public long FailedRequests { get; init; }

/// <summary>
/// Requests per second
/// </summary>
public double RequestsPerSecond { get; init; }

/// <summary>
/// Average response time in ms
/// </summary>
public double AverageResponseTime { get; init; }

/// <summary>
/// Minimum response time in ms
/// </summary>
public double MinResponseTime { get; init; }

/// <summary>
/// Maximum response time in ms
/// </summary>
public double MaxResponseTime { get; init; }

/// <summary>
/// 50th percentile response time
/// </summary>
public double Percentile50 { get; init; }

/// <summary>
/// 90th percentile response time
/// </summary>
public double Percentile90 { get; init; }

/// <summary>
/// 95th percentile response time
/// </summary>
public double Percentile95 { get; init; }

/// <summary>
/// 99th percentile response time
/// </summary>
public double Percentile99 { get; init; }
}

/// <summary>
/// Performance difference between two executions
/// </summary>
public record PerformanceDelta
{
/// <summary>
/// Requests per second difference (positive = improvement)
/// </summary>
public double RpsDelta { get; init; }

/// <summary>
/// Requests per second percentage change
/// </summary>
public double RpsPercentChange { get; init; }

/// <summary>
/// Average response time difference (negative = improvement)
/// </summary>
public double AvgResponseTimeDelta { get; init; }

/// <summary>
/// Average response time percentage change
/// </summary>
public double AvgResponseTimePercentChange { get; init; }

/// <summary>
/// Minimum response time difference
/// </summary>
public double MinResponseTimeDelta { get; init; }

/// <summary>
/// Maximum response time difference
/// </summary>
public double MaxResponseTimeDelta { get; init; }

/// <summary>
/// P50 response time difference
/// </summary>
public double P50Delta { get; init; }

/// <summary>
/// P50 percentage change
/// </summary>
public double P50PercentChange { get; init; }

/// <summary>
/// P90 response time difference
/// </summary>
public double P90Delta { get; init; }

/// <summary>
/// P90 percentage change
/// </summary>
public double P90PercentChange { get; init; }

/// <summary>
/// P95 response time difference
/// </summary>
public double P95Delta { get; init; }

/// <summary>
/// P95 percentage change
/// </summary>
public double P95PercentChange { get; init; }

/// <summary>
/// P99 response time difference
/// </summary>
public double P99Delta { get; init; }

/// <summary>
/// P99 percentage change
/// </summary>
public double P99PercentChange { get; init; }

/// <summary>
/// Failure rate difference
/// </summary>
public double FailureRateDelta { get; init; }

/// <summary>
/// Overall assessment of performance change
/// </summary>
public string Assessment { get; init; } = string.Empty;
}
113 changes: 113 additions & 0 deletions src/NFury/Web/Services/ExecutionService.cs
Original file line number Diff line number Diff line change
Expand Up @@ -697,6 +697,119 @@ public static TestExecution MapExecution(SqliteDataReader reader)
ErrorMessage = reader.IsDBNull(25) ? null : reader.GetString(25)
};
}

/// <summary>
/// Compares two test executions and calculates performance delta
/// </summary>
/// <param name="baselineId">The baseline execution ID</param>
/// <param name="compareId">The execution ID to compare against baseline</param>
/// <returns>The comparison result or null if either execution is not found</returns>
public async Task<ExecutionComparisonResult?> CompareExecutionsAsync(int baselineId, int compareId)
{
var baseline = await GetExecutionByIdAsync(baselineId);
var compare = await GetExecutionByIdAsync(compareId);

if (baseline == null || compare == null)
return null;

var baselineSummary = new ExecutionSummary
{
Id = baseline.Id,
TestId = baseline.TestId,
EndpointName = baseline.Endpoint?.Name,
Url = baseline.Url,
StartedAt = baseline.StartedAt,
TotalRequests = baseline.TotalRequests,
FailedRequests = baseline.FailedRequests,
RequestsPerSecond = baseline.RequestsPerSecond,
AverageResponseTime = baseline.AverageResponseTime,
MinResponseTime = baseline.MinResponseTime,
MaxResponseTime = baseline.MaxResponseTime,
Percentile50 = baseline.Percentile50,
Percentile90 = baseline.Percentile90,
Percentile95 = baseline.Percentile95,
Percentile99 = baseline.Percentile99
};

var compareSummary = new ExecutionSummary
{
Id = compare.Id,
TestId = compare.TestId,
EndpointName = compare.Endpoint?.Name,
Url = compare.Url,
StartedAt = compare.StartedAt,
TotalRequests = compare.TotalRequests,
FailedRequests = compare.FailedRequests,
RequestsPerSecond = compare.RequestsPerSecond,
AverageResponseTime = compare.AverageResponseTime,
MinResponseTime = compare.MinResponseTime,
MaxResponseTime = compare.MaxResponseTime,
Percentile50 = compare.Percentile50,
Percentile90 = compare.Percentile90,
Percentile95 = compare.Percentile95,
Percentile99 = compare.Percentile99
};

var rpsDelta = compare.RequestsPerSecond - baseline.RequestsPerSecond;
var avgResponseDelta = compare.AverageResponseTime - baseline.AverageResponseTime;
var p50Delta = compare.Percentile50 - baseline.Percentile50;
var p90Delta = compare.Percentile90 - baseline.Percentile90;
var p95Delta = compare.Percentile95 - baseline.Percentile95;
var p99Delta = compare.Percentile99 - baseline.Percentile99;

var baselineFailureRate = baseline.TotalRequests > 0 ? (double)baseline.FailedRequests / baseline.TotalRequests : 0;
var compareFailureRate = compare.TotalRequests > 0 ? (double)compare.FailedRequests / compare.TotalRequests : 0;

var assessment = DetermineAssessment(rpsDelta, avgResponseDelta, baseline.RequestsPerSecond, baseline.AverageResponseTime);

var delta = new PerformanceDelta
{
RpsDelta = rpsDelta,
RpsPercentChange = baseline.RequestsPerSecond > 0 ? (rpsDelta / baseline.RequestsPerSecond) * 100 : 0,
AvgResponseTimeDelta = avgResponseDelta,
AvgResponseTimePercentChange = baseline.AverageResponseTime > 0 ? (avgResponseDelta / baseline.AverageResponseTime) * 100 : 0,
MinResponseTimeDelta = compare.MinResponseTime - baseline.MinResponseTime,
MaxResponseTimeDelta = compare.MaxResponseTime - baseline.MaxResponseTime,
P50Delta = p50Delta,
P50PercentChange = baseline.Percentile50 > 0 ? (p50Delta / baseline.Percentile50) * 100 : 0,
P90Delta = p90Delta,
P90PercentChange = baseline.Percentile90 > 0 ? (p90Delta / baseline.Percentile90) * 100 : 0,
P95Delta = p95Delta,
P95PercentChange = baseline.Percentile95 > 0 ? (p95Delta / baseline.Percentile95) * 100 : 0,
P99Delta = p99Delta,
P99PercentChange = baseline.Percentile99 > 0 ? (p99Delta / baseline.Percentile99) * 100 : 0,
FailureRateDelta = (compareFailureRate - baselineFailureRate) * 100,
Assessment = assessment
};

return new ExecutionComparisonResult
{
Baseline = baselineSummary,
Compare = compareSummary,
Delta = delta
};
}

private static string DetermineAssessment(double rpsDelta, double avgResponseDelta, double baselineRps, double baselineAvgResponse)
{
var rpsPercentChange = baselineRps > 0 ? (rpsDelta / baselineRps) * 100 : 0;
var responsePercentChange = baselineAvgResponse > 0 ? (avgResponseDelta / baselineAvgResponse) * 100 : 0;

var rpsImproved = rpsPercentChange > 5;
var rpsRegressed = rpsPercentChange < -5;
var responseImproved = responsePercentChange < -5;
var responseRegressed = responsePercentChange > 5;

if (rpsImproved && responseImproved)
return "Significant Improvement";
if (rpsImproved || responseImproved)
return "Improvement";
if (rpsRegressed && responseRegressed)
return "Significant Regression";
if (rpsRegressed || responseRegressed)
return "Regression";
return "No Significant Change";
}
Comment on lines +793 to +812
Copy link

Copilot AI Jan 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The DetermineAssessment method uses hardcoded threshold values of 5% without explanation or configuration. These thresholds may be too high or too low depending on the use case. Consider making these thresholds configurable, or at least document why 5% was chosen as the threshold for significant change. Additionally, the logic doesn't account for the failure rate delta, which could be a critical indicator of regression.

Copilot uses AI. Check for mistakes.
}

/// <summary>
Expand Down
Loading