Added aggregations and GROUP BY

[liuzicheng1987]

No description provided.

[gemini-code-assist]

Hello @liuzicheng1987, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

Summary of Changes

In this pull request, Gemini has added support for SQL aggregations and the GROUP BY clause to the sqlgen library. This significantly enhances the query capabilities, allowing users to perform complex analytical queries directly from C++ in a type-safe manner. The changes involve a substantial refactoring of the select_from query building mechanism to accommodate the selection of arbitrary fields (columns, aggregations, literal values) and the introduction of new operators like group_by and to for specifying grouping criteria and result types/containers. Compile-time checks have been added to ensure correct usage of aggregations and GROUP BY clauses. Comprehensive documentation and tests for both PostgreSQL and SQLite backends are included.

Highlights

• SQL Aggregations: Introduced support for common SQL aggregation functions: count(), count_distinct(col), avg(col), sum(col), max(col), and min(col).
• GROUP BY Clause: Added the group_by(...) operator to specify columns for grouping results in SELECT queries.
• Flexible SELECT Fields: Refactored select_from to allow selecting arbitrary expressions, including columns, aggregations, and literal values, instead of being limited to mapping directly to a struct's fields.
• Aliasing Fields: Introduced the .as<"alias">() member function for columns and aggregations, and the as<"alias">(value) function/| as<"alias"> operator for aliasing selected fields.
• Result Type Specification (to<T>): Added the to<T> operator to explicitly specify the desired result type or container (e.g., std::vector<MyStruct>) for a select_from query, enabling mapping to custom structs that match the selected fields by name.
• Compile-Time Checks: Implemented static assertions to validate the correct usage of aggregations and GROUP BY clauses, ensuring that selected non-aggregated columns are included in the GROUP BY clause when aggregations are present.
• Documentation and Tests: Added new documentation pages explaining group_by and aggregations, and included new tests for both PostgreSQL and SQLite covering basic aggregations, aggregations with nullable columns, and GROUP BY.
• Internal Refactoring: Made internal changes to the transpilation logic (transpilation/) and dynamic query representation (dynamic/) to support the new features, including handling aliased fields and different types of selected expressions.
• Memory Management: Modified the result reading logic (internal/from_str_vec.hpp) to use placement new and manual destructor calls for potentially better memory management when constructing result objects.

Changelog

Click here to see the changelog

• README.md
  • Added a new section 'Grouping and Aggregating Data' with a C++ example and generated SQL.
• docs/README.md
  • Added a link to the new documentation page for group_by and aggregations.
• docs/group_by_and_aggregations.md
  • New documentation file detailing the usage of sqlgen::group_by and aggregation functions.
  • Includes examples for basic aggregations and aggregations with GROUP BY.
  • Explains how nullable values are handled in aggregations.
  • Documents the .as operator for aliasing.
• include/sqlgen.hpp
  • Included new header files for aggregations, aliasing (as), group_by, select_from, and to.
• include/sqlgen/aggregations.hpp
  • New header defining C++ wrappers for SQL aggregation functions (avg, count, count_distinct, max, min, sum).
• include/sqlgen/as.hpp
  • New header defining the As struct and as constant for aliasing.
• include/sqlgen/col.hpp
  • Added an as() member function to the Col struct for aliasing columns.
• include/sqlgen/dynamic/Aggregation.hpp
  • New header defining dynamic representation for SQL aggregation functions.
• include/sqlgen/dynamic/ColumnOrAggregation.hpp
  • New header defining a tagged union for dynamic columns or aggregations.
• include/sqlgen/dynamic/GroupBy.hpp
  • New header defining dynamic representation for the GROUP BY clause.
• include/sqlgen/dynamic/SelectFrom.hpp
  • Modified the dynamic SelectFrom struct to use a vector of Field (supporting columns, aggregations, values, and aliases) instead of just columns.
  • Added an optional group_by member.
• include/sqlgen/group_by.hpp
  • New header defining the GroupBy struct and group_by function.
  • Includes static assertions for correct clause ordering and usage.
• include/sqlgen/internal/GetColType.hpp
  • New internal helper for getting column types.
• include/sqlgen/internal/call_destructors_where_necessary.hpp
  • New internal helper for manual destructor calls.
• include/sqlgen/internal/from_str_vec.hpp
  • Included call_destructors_where_necessary.hpp.
  • Modified result reading to use placement new and manual destructor calls.
• include/sqlgen/limit.hpp
  • Added support for applying limit to SelectFrom queries.
  • Added static assertions for correct clause ordering.
• include/sqlgen/order_by.hpp
  • Added support for applying order_by to SelectFrom queries.
  • Added static assertions for correct clause ordering.
  • Improved static assertion message.
• include/sqlgen/read.hpp
  • Changed transpilation function for Read to read_to_select_from.
  • Removed order_by_ member from Read struct.
• include/sqlgen/select_from.hpp
  • New header defining the core SelectFrom struct and select_from function.
  • Implements the query building logic with support for fields, where, group by, order by, limit, and mapping to result types via to<T>.
• include/sqlgen/to.hpp
  • New header defining the To struct and to constant for specifying result types.
• include/sqlgen/transpilation/As.hpp
  • New header defining the transpilation struct for aliasing.
• include/sqlgen/transpilation/aggregations.hpp
  • New header defining transpilation structs for aggregation functions.
• include/sqlgen/transpilation/check_aggregations.hpp
  • New header implementing compile-time checks for aggregation and group by usage.
• include/sqlgen/transpilation/fields_to_named_tuple_t.hpp
  • New internal helper for determining the result named tuple type.
• include/sqlgen/transpilation/group_by_t.hpp
  • New header defining the transpilation struct for GROUP BY.
  • Includes static assertion for column existence.
• include/sqlgen/transpilation/is_nullable.hpp
  • Modified is_nullable_v to handle cvref qualifiers.
• include/sqlgen/transpilation/make_field.hpp
  • New internal helper for creating dynamic fields from user input.
  • Includes static assertions for valid aggregation arguments.
• include/sqlgen/transpilation/make_fields.hpp
  • New internal helper for creating a vector of dynamic fields.
• include/sqlgen/transpilation/order_by_t.hpp
  • Improved static assertion message.
• include/sqlgen/transpilation/read_to_select_from.hpp
  • New internal helper for transpiling Read operations to dynamic SelectFrom.
• include/sqlgen/transpilation/remove_as_t.hpp
  • New internal helper for removing the As wrapper type.
• include/sqlgen/transpilation/remove_nullable_t.hpp
  • New internal helper for removing nullable wrappers.
• include/sqlgen/transpilation/to_group_by.hpp
  • New internal helper for transpiling GroupBy to dynamic GroupBy.
• include/sqlgen/transpilation/to_select_from.hpp
  • Modified the main transpilation function for SelectFrom to handle the new field structure and GROUP BY.
  • Included check_aggregations static assertion.
• include/sqlgen/transpilation/to_sql.hpp
  • Added support for transpiling the new SelectFrom struct.
  • Updated Read transpilation to use read_to_select_from.
• include/sqlgen/where.hpp
  • Added support for applying where to SelectFrom queries.
  • Added static assertions for correct clause ordering.
• src/sqlgen/postgres/to_sql.cpp
  • Implemented SQL generation for dynamic aggregations.
  • Implemented SQL generation for the new dynamic SelectFrom structure, including fields, aliases, and GROUP BY.
• src/sqlgen/sqlite/to_sql.cpp
  • Implemented SQL generation for dynamic aggregations.
  • Implemented SQL generation for the new dynamic SelectFrom structure, including fields, aliases, and GROUP BY.
• tests/postgres/test_aggregations.cpp
  • New test file for basic PostgreSQL aggregations.
• tests/postgres/test_aggregations_with_nullable.cpp
  • New test file for PostgreSQL aggregations with nullable columns.
• tests/postgres/test_delete_from_dry.cpp
  • Removed unused include.
• tests/postgres/test_drop_dry.cpp
  • Removed unused include.
• tests/postgres/test_group_by.cpp
  • New test file for PostgreSQL GROUP BY with aggregations.
• tests/postgres/test_range_select_from.cpp
  • New test file for PostgreSQL select_from returning a range.
• tests/postgres/test_select_from_with_to.cpp
  • New test file for PostgreSQL select_from using to<std::vector<T>>.
• tests/postgres/test_to_select_from2_dry.cpp
  • New test file for dry run SQL generation of a complex select_from query.
• tests/postgres/test_to_select_from_dry.cpp
  • Removed unused include.
• tests/sqlite/test_aggregations.cpp
  • New test file for basic SQLite aggregations.
• tests/sqlite/test_aggregations_with_nullable.cpp
  • New test file for SQLite aggregations with nullable columns.
• tests/sqlite/test_group_by.cpp
  • New test file for SQLite GROUP BY with aggregations.
• tests/sqlite/test_range_select_from.cpp
  • New test file for SQLite select_from returning a range.
• tests/sqlite/test_range_select_from_with_to.cpp
  • New test file for SQLite select_from using to<std::vector<T>>.
• tests/sqlite/test_to_select_from.cpp
  • Updated dry run test to use read_to_select_from.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces valuable aggregation and GROUP BY functionalities, significantly enhancing the query capabilities of sqlgen. The implementation leverages C++ templates effectively for type safety and provides good compile-time checks. The new documentation is also helpful for users to understand these new features.

Overall, the changes are well-structured. I've identified a couple of areas for potential refinement, particularly concerning memory management in from_str_vec.hpp and a potentially unused function in the SQLite SQL generation.

Summary of Findings

• Memory Management in from_str_vec.hpp: The use of manual memory management techniques (placement new, manual destructors) in from_str_vec.hpp is complex and warrants careful review to ensure correctness and understand the driving factors.
• Potentially Unused Code: The function column_or_agg_to_str in src/sqlgen/sqlite/to_sql.cpp appears to be unused in the current diff and might be redundant.
• Documentation Example Clarity: In docs/group_by_and_aggregations.md (line 75), the example count_distinct("last_name"_c) in a query already grouped by last_name will always yield 1. While technically correct, a different example might be more illustrative for count_distinct in grouped queries. This is a minor point and was not commented on directly due to review settings.
• Static Assert Message Phrasing: The phrasing of the static_assert message in include/sqlgen/order_by.hpp (lines 24, 46) regarding limit and order_by order is slightly awkward but its logical assertion is correct. This is a minor point and was not commented on directly due to review settings.

Merge Readiness

The pull request is a strong step forward for sqlgen. However, due to the medium-severity comments regarding the memory management changes in from_str_vec.hpp and the potentially unused code in the SQLite SQL generation, I recommend addressing or clarifying these points before merging. I am not authorized to approve pull requests, so please ensure these changes are also reviewed and approved by other maintainers.