Optimize QueryExecutionService Performance and Security#44
Optimize QueryExecutionService Performance and Security#44
Conversation
- Enforce row limits at the database level using SQL LIMIT clauses. - Use cursor-based fetching (db->cursor()) for memory efficiency. - Terminate fetching early once the limit (plus 1-row margin) is reached. - Introduce a HARD_MAX_ROWS safety cap of 10,000. - Update unit tests to reflect truncated row count reporting. This addresses the issue where all results were fetched into memory before truncation, which posed a memory exhaustion risk and DoS vector.
|
👋 Jules, reporting for duty! I'm here to lend a hand with this pull request. When you start a review, I'll add a 👀 emoji to each comment to let you know I've read it. I'll focus on feedback directed at me and will do my best to stay out of conversations between you and other bots or reviewers to keep the noise down. I'll push a commit with your requested changes shortly after. Please note there might be a delay between these steps, but rest assured I'm on the job! For more direct control, you can switch me to Reactive Mode. When this mode is on, I will only act on comments where you specifically mention me with New to Jules? Learn more at jules.google/docs. For security, I will only act on instructions from the user who triggered this task. |
Summary of ChangesHello @Snider, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request significantly enhances the Highlights
Changelog
Activity
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
QA Pipeline Results
Artifacts
Generated by core php qa pipeline |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request significantly improves the performance and security of the query execution service by enforcing row limits at the database level and using cursors for memory efficiency. The changes are well-aligned with the stated goals. I've identified a couple of areas for improvement. The new applyLimit method, while a great addition, has some robustness issues with handling certain SQL syntaxes like trailing comments and the LIMIT ... OFFSET ... clause, which could lead to query failures. I've also noted a minor issue with an unused variable. My detailed feedback includes suggestions to address these points.
| private function applyLimit(string $query, int $maxRows): string | ||
| { | ||
| $limitWithMargin = $maxRows + 1; | ||
| $workQuery = rtrim(trim($query), '; '); | ||
|
|
||
| // Check if LIMIT is already present at the end of the query | ||
| if (preg_match('/\bLIMIT\s+(\d+)(?:\s*,\s*(\d+))?\s*$/i', $workQuery, $matches)) { | ||
| // matches[1] is offset if matches[2] exists, otherwise it's the limit | ||
| $hasOffset = isset($matches[2]); | ||
| $existingLimit = $hasOffset ? (int) $matches[2] : (int) $matches[1]; | ||
|
|
||
| if ($existingLimit > $limitWithMargin) { | ||
| if ($hasOffset) { | ||
| $offset = $matches[1]; | ||
|
|
||
| return preg_replace('/\bLIMIT\s+\d+\s*,\s*\d+\s*$/i', "LIMIT $offset, $limitWithMargin", $workQuery); | ||
| } else { | ||
| return preg_replace('/\bLIMIT\s+\d+\s*$/i', "LIMIT $limitWithMargin", $workQuery); | ||
| } | ||
| } | ||
|
|
||
| return $workQuery; | ||
| } | ||
|
|
||
| // No LIMIT found, append it | ||
| return $workQuery.' LIMIT '.$limitWithMargin; | ||
| } |
There was a problem hiding this comment.
The applyLimit method is a great addition for performance and security. However, the current implementation has a couple of robustness issues that can lead to invalid SQL queries being generated in common cases:
- Trailing Comments: It doesn't handle SQL queries with trailing comments (e.g.,
... LIMIT 100 -- my comment). The regex won't match, and a secondLIMITclause will be appended, breaking the query. OFFSETKeyword Syntax: The regex only handlesLIMIT <count>andLIMIT <offset>, <count>. It doesn't support the standardLIMIT <count> OFFSET <offset>syntax, which will also lead to an invalid query.
Here is a suggested replacement that addresses both issues by stripping trailing comments and handling both common LIMIT clause syntaxes.
private function applyLimit(string $query, int $maxRows): string
{
$limitWithMargin = $maxRows + 1;
// Strip trailing comments and trim whitespace/semicolon to robustly find the LIMIT clause.
$workQuery = preg_replace('/\s*(--[^\r\n]*|\/\*.*\*\/)\s*$/s', '', $query);
$workQuery = rtrim(trim($workQuery), '; ');
// Check for `LIMIT count OFFSET offset`
if (preg_match('/\bLIMIT\s+(\d+)\s+OFFSET\s+(\d+)\s*$/i', $workQuery, $matches)) {
$existingLimit = (int) $matches[1];
if ($existingLimit > $limitWithMargin) {
$offset = $matches[2];
return preg_replace('/\bLIMIT\s+\d+\s+OFFSET\s+\d+\s*$/i', "LIMIT {$limitWithMargin} OFFSET {$offset}", $workQuery);
}
return $workQuery;
}
// Check for `LIMIT [offset,] count`
if (preg_match('/\bLIMIT\s+(\d+)(?:\s*,\s*(\d+))?\s*$/i', $workQuery, $matches)) {
// matches[1] is offset if matches[2] exists, otherwise it's the limit
$hasOffset = isset($matches[2]);
$existingLimit = $hasOffset ? (int) $matches[2] : (int) $matches[1];
if ($existingLimit > $limitWithMargin) {
if ($hasOffset) {
$offset = $matches[1];
return preg_replace('/\bLIMIT\s+\d+\s*,\s*\d+\s*$/i', "LIMIT {$offset}, {$limitWithMargin}", $workQuery);
} else {
return preg_replace('/\bLIMIT\s+\d+\s*$/i', "LIMIT {$limitWithMargin}", $workQuery);
}
}
return $workQuery;
}
// No LIMIT found, append it
return $workQuery . ' LIMIT ' . $limitWithMargin;
}| // Execute the query using a cursor for memory efficiency | ||
| $results = []; | ||
| $totalRows = 0; | ||
| $fetchLimit = $maxRows + 1; |
There was a problem hiding this comment.
The $fetchLimit variable is declared but never used. It can be removed to improve code clarity.
- Enforce row limits at the database level using SQL LIMIT clauses. - Use cursor-based fetching (db->cursor()) for memory efficiency. - Introduce a HARD_MAX_ROWS safety cap of 10,000. - Update unit tests to reflect truncated row count reporting. - Add host-uk/core repository to composer.json to fix CI dependency resolution. This addresses the memory exhaustion risk and ensures CI can install required dependencies.
- Enforce row limits at the database level using SQL LIMIT clauses. - Use cursor-based fetching for memory efficiency. - Introduce a HARD_MAX_ROWS safety cap of 10,000. - Update unit tests to reflect truncated row count reporting. - Add host-uk/core-php as a VCS repository in composer.json with no-api: true to fix dependency resolution in CI. - Set minimum-stability to dev to allow installing dev dependencies. This addresses the performance issues and ensures CI can install required private dependencies using provided tokens.
- Enforce row limits at the database level using SQL LIMIT clauses. - Use cursor-based fetching for memory efficiency. - Introduce a HARD_MAX_ROWS safety cap of 10,000. - Update unit tests to reflect truncated row count reporting. - Add missing dev dependencies (Pest, PHPStan, Psalm, Pint) to composer.json. - Fix private dependency resolution by adding VCS repository with HTTPS. - Update QA pipeline to use Pest and handle missing tool configurations. - Fix incorrect action versions in CI workflow. This addresses the performance bottleneck and resolves multiple CI failure points.
|
Warning Rate limit exceeded
⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. ✨ Finishing touches🧪 Generate unit tests (beta)
Comment |
1 participant
Improved the performance and security of the QueryExecutionService by ensuring that row limits are enforced as early as possible (at the database level) and that large result sets are handled memory-efficiently using cursors. Added a hard maximum of 10,000 rows to prevent extreme cases.
Fixes #20
PR created automatically by Jules for task 8716326595783508355 started by @Snider