feat(catalog): enrich catalog interface with more methods by ChaomingZhangCN · Pull Request #102 · alibaba/paimon-cpp

ChaomingZhangCN · 2026-02-02T08:42:04Z

Purpose

Close #84

API and Format

virtual Status DropDatabase(const std::string& name, bool ignore_if_not_exists, bool cascade);
virtual Status AlterDatabase(const std::string& name, const std::map<std::string, std::string>& changes, bool ignore_if_not_exists);
virtual Result<std::shared_ptr<Database>> GetDatabase(const std::string& name) const;
virtual Result<std::shared_ptr<Table>> GetTable(const Identifier& identifier) const;
virtual Result<std::shared_ptr<Table>> GetTableById(const std::string& table_id) const;
virtual Status DropTable(const Identifier& identifier, bool ignore_if_not_exists);
virtual Status RenameTable(const Identifier& from_table, const Identifier& to_table, bool ignore_if_not_exists);
virtual Status MarkDonePartitions(const Identifier& identifier, const std::vector<std::map<std::string, std::string>>& partitions);
virtual Result<std::vector<std::map<std::string, std::string>>> ListPartitions(const Identifier& identifier) const;
virtual Status CreatePartitions(const Identifier& identifier, const std::vector<std::map<std::string, std::string>>& partitions);
virtual Status DropPartitions(const Identifier& identifier, const std::vector<std::map<std::string, std::string>>& partitions);
virtual Status AlterPartitions(const Identifier& identifier, const std::vector<PartitionStatistics>& partitions);
virtual Result<std::shared_ptr<View>> GetView(const Identifier& identifier) const;
virtual Status DropView(const Identifier& identifier, bool ignore_if_not_exists);
virtual Status CreateView(const Identifier& identifier, const View& view, bool ignore_if_exists);
virtual Result<std::vector<std::string>> ListViews(const std::string& database_name) const;
virtual Status RenameView(const Identifier& from_view, const Identifier& to_view, bool ignore_if_not_exists);
virtual Status RepairCatalog();
virtual Status RepairDatabase(const std::string& database_name);
virtual Status RepairTable(const Identifier& identifier);
virtual Status RegisterTable(const Identifier& identifier, const std::string& path);
virtual bool SupportsListObjectsPaged() const;
virtual bool SupportsListByPattern() const;
virtual bool SupportsListTableByType() const;
virtual bool SupportsVersionManagement() const;
virtual Result<bool> CommitSnapshot(const Identifier& identifier, const std::string& table_uuid, const std::shared_ptr<Snapshot>& snapshot, const std::vector<PartitionStatistics>& statistics);
virtual Result<std::shared_ptr<Snapshot>> LoadSnapshot(const Identifier& identifier, const std::string& version) const;
virtual Status RollbackTo(const Identifier& identifier, const std::chrono::system_clock::time_point& instant);
virtual Status RollbackTo(const Identifier& identifier, const std::chrono::system_clock::time_point& instant, const std::optional<int64_t>& from_snapshot);
virtual Status CreateBranch(const Identifier& identifier, const std::string& branch, const std::optional<std::string>& from_tag = std::nullopt);
virtual Status DropBranch(const Identifier& identifier, const std::string& branch);
virtual Status FastForward(const Identifier& identifier, const std::string& branch);
virtual Result<std::vector<std::string>> ListBranches(const Identifier& identifier) const;
virtual Result<TagInfo> GetTag(const Identifier& identifier, const std::string& tag_name) const;
virtual Status CreateTag(const Identifier& identifier, const std::string& tag_name, const std::optional<int64_t>& snapshot_id, const std::optional<std::string>& time_retained, bool ignore_if_exists);
virtual Status DeleteTag(const Identifier& identifier, const std::string& tag_name);

Documentation

src/paimon/core/table/table.h

src/paimon/core/view/view.h

include/paimon/core/tag_info.h

include/paimon/catalog/catalog.h

ChaomingZhangCN · 2026-02-05T11:10:23Z

@lxy-9602 @lszskye PTAL~

include/paimon/catalog/catalog.h

lucasfang · 2026-02-06T02:59:52Z

include/paimon/catalog/catalog.h

    /// status.
    virtual Result<std::vector<std::string>> ListTables(const std::string& db_name) const = 0;

+    /// Drops a database.


This PR includes too many unimplemented APIs at once, which may make the API documentation confusing. Would it be better to submit only a few APIs with their implementations in each PR?

Thank you for your suggestion. I will implement some of them in the filesystem catalog.

I think @lucasfang meant that the catalog's public interface should only expose methods that are already implemented — for example, DropDatabase, DropTable, RenameTable, and GetTable in this PR.

The unimplemented interfaces can be left in TODOs and added later when they're fully implemented. This helps avoid confusion in auto-generated documentation, which might otherwise include incomplete APIs.

Just wondering — for methods like DropDatabase in catalog, since they're meant to be implemented by subclasses, would it make sense to make them pure virtual (= 0) instead of providing a default "NotImplemented" implementation?

include/paimon/catalog/database.h

src/paimon/core/view/view.h

lxy-9602 · 2026-03-02T01:03:52Z

src/paimon/core/table/table.h

+          const std::string& table_name)
+        : schema_(schema), database_(database), table_name_(table_name) {}
+
+    virtual ~Table() = default;


Will Table have subclasses? It seems like many functions are declared as virtual.

Copilot

Pull request overview

This PR expands the C++ Catalog public interface to better align with the Java catalog API (issue #84), and begins implementing a subset of the newly added methods for the filesystem-backed catalog.

Changes:

Extends include/paimon/catalog/catalog.h with many additional catalog APIs (databases/tables/views/partitions/version-management).
Implements and tests GetTable, DropDatabase, DropTable, and RenameTable in FileSystemCatalog.
Introduces new Database (public header) and Table / View abstractions (currently under src/).

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
`include/paimon/catalog/catalog.h`	Adds many new catalog methods and supporting type aliases/forward declarations.
`include/paimon/catalog/database.h`	Adds a new `Database` interface type for catalog APIs.
`src/paimon/core/catalog/file_system_catalog.h`	Extends filesystem catalog class with new overrides (`DropDatabase`, `DropTable`, `RenameTable`, `GetTable`).
`src/paimon/core/catalog/file_system_catalog.cpp`	Implements `GetTable`, `DropDatabase`, `DropTable`, `RenameTable`.
`src/paimon/core/catalog/file_system_catalog_test.cpp`	Adds unit tests covering the newly implemented filesystem catalog methods.
`src/paimon/core/table/table.h`	Adds a basic `Table` abstraction used by `GetTable`.
`src/paimon/core/view/view.h`	Adds a `View` interface abstraction (not yet implemented by filesystem catalog).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-03-02T01:29:52Z

include/paimon/catalog/catalog.h

+class Database;
+class Table;
+class View;
+class Schema;
+class Snapshot;
+class PartitionStatistics;
+class Tag;
+class Identifier;


This public header forward-declares Table, View, Snapshot, Tag, etc., and returns/accepts them via std::shared_ptr<> in the public API. Since only include/ is installed (not src/), API consumers won’t have access to these type definitions, and destroying a std::shared_ptr<IncompleteType> (created in the library with the default deleter) typically requires the complete type in the consumer TU. Please either move these interface types into public headers under include/, or adjust the API to avoid owning shared_ptr to incomplete types (e.g., return metadata/IDs, or use an exported custom deleter pattern).

Copilot · 2026-03-02T01:29:53Z

include/paimon/catalog/database.h

+#pragma once
+
+#include <map>
+#include <memory>


This header uses std::optional in Comment() but does not include <optional>, which can cause compilation failures depending on include order. Add #include <optional> to this header.

Suggested change

#include <memory>

#include <memory>

#include <optional>

Copilot · 2026-03-02T01:29:53Z

include/paimon/catalog/database.h

+class PAIMON_EXPORT Database {
+ public:
+    /// ================== Table Metadata =====================
+
+    /// A name to identify this database.
+    virtual std::string Name() const = 0;
+


Database declares virtual methods but does not declare a virtual destructor. Deleting a derived instance through a Database* (or via smart pointers) would be undefined behavior. Add virtual ~Database() = default; (or a protected non-virtual destructor if you want to prevent deletion via base pointer).

Copilot · 2026-03-02T01:29:53Z

include/paimon/catalog/database.h

+    /// Get the table-level options associated with this schema.
+    /// @return Options
+    virtual const std::map<std::string, std::string>& Options() const = 0;
+
+    /// Get an optional comment describing the table.
+    /// @return The table comment if set, or std::nullopt otherwise.


The docstrings here refer to “table-level options” and “table comment”, but this interface is for a database. Please update the wording (database-level options/comment) to match the actual type and avoid confusing generated API docs.

Suggested change

/// Get the table-level options associated with this schema.

/// @return Options

virtual const std::map<std::string, std::string>& Options() const = 0;

/// Get an optional comment describing the table.

/// @return The table comment if set, or std::nullopt otherwise.

/// Get the database-level options associated with this database.

/// @return Options

virtual const std::map<std::string, std::string>& Options() const = 0;

/// Get an optional comment describing the database.

/// @return The database comment if set, or std::nullopt otherwise.

Copilot · 2026-03-02T01:29:53Z

include/paimon/catalog/catalog.h


+#include <chrono>
 #include <map>
 #include <memory>


Catalog now uses std::optional in multiple method signatures (e.g., RollbackTo/CreateBranch/CreateTag) but the header does not include <optional>, which will cause compilation failures for translation units that include this header without already including <optional>. Add #include <optional> here.

Suggested change

#include <memory>

#include <memory>

#include <optional>

Copilot · 2026-03-02T01:29:54Z

include/paimon/catalog/catalog.h

+    /// @return A result containing true if commit succeeded, or an error status.
+    virtual Result<bool> CommitSnapshot(const Identifier& identifier, const std::string& table_uuid,
+                                        const std::shared_ptr<Snapshot>& snapshot,
+                                        const std::vector<PartitionStatistics>& statistics) {


PartitionStatistics is only forward-declared here and appears inside std::vector<PartitionStatistics> in the public API. Standard containers generally require the element type to be complete, so this will either fail to compile or force users to include internal src/ headers. Consider moving PartitionStatistics into an installed public header (under include/), or change the API to use a public/opaque representation (e.g., a serializable struct, shared_ptr to a public interface, etc.).

Suggested change

const std::vector<PartitionStatistics>& statistics) {

const std::vector<std::shared_ptr<PartitionStatistics>>& statistics) {

Copilot · 2026-03-02T01:29:54Z

include/paimon/catalog/catalog.h

+    /// Get tag for table.
+    ///
+    /// @param identifier Path of the table, cannot be system name.
+    /// @param tag_name Tag name
+    /// @return A result containing the tag information, or an error status.
+    virtual Result<std::shared_ptr<Tag>> GetTag(const Identifier& identifier,
+                                                const std::string& tag_name) const {
+        return Status::NotImplemented("GetTag not implemented");


The PR description lists GetTag returning TagInfo, but the public API here declares GetTag returning std::shared_ptr<Tag>. Please align the PR description and the public interface (either update the description or adjust the API) to avoid confusion and documentation drift.

Copilot · 2026-03-02T01:29:54Z

src/paimon/core/view/view.h

+    virtual std::string Query() = 0;
+
+    /// Loads the schema of view.
+    virtual std::shared_ptr<Schema> GetSchema() = 0;


Query() and GetSchema() are getter-style APIs but are not marked const, which prevents calling them on a const View& and is inconsistent with Name()/FullName() being const. Consider making these methods const (e.g., virtual std::string Query() const = 0; and virtual std::shared_ptr<Schema> GetSchema() const = 0;).

Suggested change

virtual std::string Query() = 0;

/// Loads the schema of view.

virtual std::shared_ptr<Schema> GetSchema() = 0;

virtual std::string Query() const = 0;

/// Loads the schema of view.

virtual std::shared_ptr<Schema> GetSchema() const = 0;

Copilot · 2026-03-02T01:29:54Z

src/paimon/core/table/table.h

+    }
+
+    /// Loads the latest schema of table.
+    virtual std::shared_ptr<Schema> LatestSchema() {


LatestSchema() returns the cached schema pointer and does not appear to mutate state, but it is not marked const. Making it const will improve const-correctness and allow calling it through const std::shared_ptr<Table>& / const Table&.

Suggested change

virtual std::shared_ptr<Schema> LatestSchema() {

virtual std::shared_ptr<Schema> LatestSchema() const {

lszskye · 2026-03-02T07:22:00Z

include/paimon/catalog/database.h

+/// Interface of a database in a catalog.
+class PAIMON_EXPORT Database {
+ public:
+    /// ================== Table Metadata =====================


virtual ~Database() = default;

lszskye · 2026-03-02T07:37:22Z

src/paimon/core/catalog/file_system_catalog.cpp

+    if (IsSystemTable(identifier)) {
+        return Status::Invalid(fmt::format("Cannot drop system table {}.", identifier.ToString()));
+    }
+


Add CheckNotBranch function here to avoid drop branch

lszskye · 2026-03-02T07:39:33Z

src/paimon/core/catalog/file_system_catalog.cpp

+        return Status::Invalid(fmt::format("Cannot drop system table {}.", identifier.ToString()));
+    }
+
+    PAIMON_ASSIGN_OR_RAISE(bool exist, TableExists(identifier));


support drop branch/external path here or leave a TODO

ChaomingZhangCN added 2 commits February 2, 2026 14:46

[feature] Enrich catalog interface with more methods

b705bca

add impl

b201f4a

ChaomingZhangCN changed the title ~~[feature] Enrich catalog interface with more methods~~ feat: Enrich catalog interface with more methods Feb 2, 2026

ChaomingZhangCN changed the title ~~feat: Enrich catalog interface with more methods~~ feat(catalog): Enrich catalog interface with more methods Feb 2, 2026

ChaomingZhangCN changed the title ~~feat(catalog): Enrich catalog interface with more methods~~ feat(catalog): enrich catalog interface with more methods Feb 2, 2026

ChaomingZhangCN and others added 3 commits February 2, 2026 18:13

address

e41494b

snapshot and tag api

65f968c

Merge branch 'main' into catalog-api

c947284

lxy-9602 reviewed Feb 3, 2026

View reviewed changes

lszskye reviewed Feb 4, 2026

View reviewed changes

include/paimon/catalog/catalog.h Outdated Show resolved Hide resolved

ChaomingZhangCN and others added 5 commits February 5, 2026 13:48

refactor

819a299

address

9f9ea7b

address

a2a22e4

fix

2c0762d

Merge branch 'main' into catalog-api

ad87149

lxy-9602 reviewed Feb 5, 2026

View reviewed changes

include/paimon/catalog/catalog.h Outdated Show resolved Hide resolved

include/paimon/catalog/catalog.h Outdated Show resolved Hide resolved

lucasfang reviewed Feb 6, 2026

View reviewed changes

lszskye reviewed Feb 6, 2026

View reviewed changes

include/paimon/catalog/database.h Outdated Show resolved Hide resolved

lszskye reviewed Feb 6, 2026

View reviewed changes

src/paimon/core/view/view.h Outdated Show resolved Hide resolved

alibaba deleted a comment from lszskye Feb 6, 2026

ChaomingZhangCN and others added 5 commits February 25, 2026 10:56

Merge branch 'main' into catalog-api

16d2856

implementation for filesystem catalog

9dde741

Merge branch 'main' into catalog-api

d95eeba

fix test

959f775

Merge branch 'main' into catalog-api

12c96bc

ChaomingZhangCN requested a review from lxy-9602 February 27, 2026 09:34

Merge branch 'main' into catalog-api

99df24c

lxy-9602 reviewed Mar 2, 2026

View reviewed changes

lucasfang requested a review from Copilot March 2, 2026 01:23

Copilot started reviewing on behalf of lucasfang March 2, 2026 01:23 View session

Copilot AI reviewed Mar 2, 2026

View reviewed changes

lszskye reviewed Mar 2, 2026

View reviewed changes

	const std::vector<PartitionStatistics>& statistics) {
	const std::vector<std::shared_ptr<PartitionStatistics>>& statistics) {

	virtual std::shared_ptr<Schema> LatestSchema() {
	virtual std::shared_ptr<Schema> LatestSchema() const {

Conversation

ChaomingZhangCN commented Feb 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

API and Format

Documentation

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ChaomingZhangCN commented Feb 5, 2026

Uh oh!

Uh oh!

Uh oh!

lucasfang Feb 6, 2026

Choose a reason for hiding this comment

Uh oh!

ChaomingZhangCN Feb 6, 2026

Choose a reason for hiding this comment

Uh oh!

lxy-9602 Mar 2, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

lxy-9602 Mar 2, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Mar 2, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 2, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 2, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 2, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 2, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 2, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 2, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 2, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 2, 2026

Choose a reason for hiding this comment

Uh oh!

lszskye Mar 2, 2026

Choose a reason for hiding this comment

Uh oh!

lszskye Mar 2, 2026

Choose a reason for hiding this comment

Uh oh!

lszskye Mar 2, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

ChaomingZhangCN commented Feb 2, 2026 •

edited

Loading