Skip to content

Refactor flow/mgt package to remove int based FK reference#1584

Open
KaveeshaPiumini wants to merge 1 commit intoasgardeo:mainfrom
KaveeshaPiumini:refactor
Open

Refactor flow/mgt package to remove int based FK reference#1584
KaveeshaPiumini wants to merge 1 commit intoasgardeo:mainfrom
KaveeshaPiumini:refactor

Conversation

@KaveeshaPiumini
Copy link
Contributor

@KaveeshaPiumini KaveeshaPiumini commented Feb 26, 2026

Purpose

This pull request refactors the flow storage schema and logic to simplify how flows and their versions are managed. The main change is the removal of the internal numeric ID (ID/FLOW_INTERNAL_ID) in favor of using the external FLOW_ID (UUID) as the primary and foreign key throughout the schema and code. This streamlines queries, reduces indirection, and simplifies both the database structure and the application logic. The test suite is also updated to cover the new logic and ensure correctness.

Database schema and query refactoring:

  • Removed the internal numeric ID and FLOW_INTERNAL_ID columns from the FLOW and FLOW_VERSION tables in both Postgres and SQLite schemas, making FLOW_ID the primary key and reference for all operations. [1] [2] [3] [4]
  • Updated all relevant SQL queries and constants to use FLOW_ID instead of the internal ID, including joins, inserts, deletes, and version management. [1] [2] [3] [4]

Application logic simplification:

  • Refactored the flow management code (store.go) to remove all logic related to fetching or using internal IDs, updating all data access to use FLOW_ID directly. This includes creation, updating, listing, restoring, and deleting flow versions. [1] [2] [3] [4] [5] [6] [7]
  • Removed now-unnecessary helper methods for retrieving internal IDs from the flow store implementation.

Test coverage improvements:

  • Added and updated unit tests in store_test.go to validate the new logic, including successful flow and version listing, retrieval, and type filtering. These tests ensure the refactored storage logic works as intended with the new schema. [1] [2] [3] [4]

Overall, this PR makes the flow storage more robust, easier to maintain, and less error-prone by eliminating unnecessary indirection and aligning the schema and code with the external flow identifier.


⚠️ Breaking Changes

🔧 Summary of Breaking Changes

The resource package has been refactored to remove all INT-based foreign key references and standardize relationships using UUID-based identifiers.

Previously, some tables referenced related entities using internal auto-incremented INT IDs. With this change, all such references have been updated to use the UUID-based identifiers of the related entities. As a result, INT-based FK columns and their associated queries have been removed or replaced.

💥 Impact

Direct database integrations, custom scripts, or downstream services that depend on the previous schema will require modification.

🔄 Migration Guide

Update database scripts to remove references to ID columns.


Related Issues

Related PRs

Checklist

  • Followed the contribution guidelines.
  • Manual test round performed and verified.
  • Documentation provided. (Add links if there are any)
    • Ran Vale and fixed all errors and warnings
  • Tests provided. (Add links if there are any)
    • Unit Tests
    • Integration Tests
  • Breaking changes. (Fill if applicable)
    • Breaking changes section filled.
    • breaking change label added.

Security checks

  • Followed secure coding standards in WSO2 Secure Coding Guidelines
  • Confirmed that this PR doesn't commit any keys, passwords, tokens, usernames, or other secrets.

Summary by CodeRabbit

  • Refactor
    • Flow and version storage now use public FLOW_ID (with deployment) as primary keys and FK targets across DB schemas and runtime, removing surrogate internal IDs; cascade delete and indexes updated for consistent lookups.
  • Tests
    • Expanded/updated tests to cover listing, fetching, versioning success paths and error handling under the FLOW_ID-based model.

@KaveeshaPiumini KaveeshaPiumini added Type/Improvement breaking change The feature/ improvement will alter the existing behaviour labels Feb 26, 2026
@coderabbitai
Copy link

coderabbitai bot commented Feb 26, 2026

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

Refactors FLOW and FLOW_VERSION to remove surrogate integer IDs and make FLOW_ID the primary key; updates schema, foreign keys, store queries, Go store implementation, and tests to operate on FLOW_ID and (FLOW_ID, DEPLOYMENT_ID) composite relations.

Changes

Cohort / File(s) Summary
Database Schema
backend/dbscripts/thunderdb/postgres.sql, backend/dbscripts/thunderdb/sqlite.sql
Removed surrogate ID columns from FLOW and FLOW_VERSION; promoted FLOW_ID to primary key; changed FLOW_VERSION PK to (FLOW_ID, VERSION, DEPLOYMENT_ID); updated FKs to reference FLOW(FLOW_ID, DEPLOYMENT_ID); dropped idx_flow_version_flow_internal_id.
Store Query Definitions
backend/internal/flow/mgt/store_constants.go
Replaced all references to FLOW_INTERNAL_ID/internal-ID with FLOW_ID in SELECT/INSERT/JOIN/WHERE queries; removed queryGetFlowInternalID.
Store Implementation
backend/internal/flow/mgt/store.go
Removed helpers that resolved internal integer IDs; updated function signatures and call sites (e.g., pushToVersionStack) to accept/use flowID (string) directly and adjusted transaction/query logic and error paths.
Store Tests
backend/internal/flow/mgt/store_test.go
Removed tests for internal-ID resolution; added/updated tests to mock and assert flows and versions using flowID (e.g., ListFlowsSuccess, GetFlowByIDSuccess, ListFlowVersionsSuccess, GetFlowVersionSuccess).

Sequence Diagram(s)

sequenceDiagram
  participant Client as Client
  participant FlowSvc as FlowService
  participant DB as Database

  rect rgba(100,149,237,0.5)
  Client->>FlowSvc: Create/Update Flow (FLOW_ID, meta)
  FlowSvc->>DB: INSERT/UPDATE FLOW (FLOW_ID, DEPLOYMENT_ID, ...)
  FlowSvc->>DB: INSERT FLOW_VERSION (FLOW_ID, VERSION, NODES, DEPLOYMENT_ID)
  DB-->>FlowSvc: OK / Constraint error
  FlowSvc-->>Client: Success / Error
  end
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related issues

Suggested reviewers

  • darshanasbg
  • ThaminduDilshan
  • rajithacharith

Poem

🐰 I swapped the numbers for a UUID bright,
FLOW_ID now leads each join and nightly flight.
Versions hop in order, no secret integer trace,
Queries sing together, migrations find their place,
A rabbit nods—schema tidy, tests alight.

🚥 Pre-merge checks | ✅ 3
✅ Passed checks (3 passed)
Check name Status Explanation
Title check ✅ Passed The title 'Refactor flow/mgt package to remove int based FK reference' clearly and concisely describes the main change—removing integer-based foreign key references in the flow management package.
Description check ✅ Passed The pull request description comprehensively covers the Purpose, Approach, Breaking Changes, Related Issues/PRs, and is well-organized with detailed explanations of the refactoring across database schema, SQL queries, application logic, and test coverage.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🧹 Nitpick comments (1)
backend/internal/flow/mgt/store_test.go (1)

156-162: Use canonical flow-type constants in type-filter tests.

Line 156 and Line 161 hardcode "authentication". Prefer string(common.FlowTypeAuthentication) to keep tests aligned with canonical enum values and avoid case drift.

♻️ Suggested test tweak
+	flowType := string(common.FlowTypeAuthentication)
 	s.mockDBProvider.EXPECT().GetConfigDBClient().Return(s.mockDBClient, nil)
-	s.mockDBClient.EXPECT().Query(queryCountFlowsWithType, "authentication", "test-deployment").
+	s.mockDBClient.EXPECT().Query(queryCountFlowsWithType, flowType, "test-deployment").
 		Return([]map[string]interface{}{{colCount: int64(1)}}, nil).Once()
-	s.mockDBClient.EXPECT().Query(queryListFlowsWithType, "authentication", "test-deployment", 10, 0).
+	s.mockDBClient.EXPECT().Query(queryListFlowsWithType, flowType, "test-deployment", 10, 0).
 		Return(flowsData, nil).Once()

-	flows, count, err := s.store.ListFlows(10, 0, "authentication")
+	flows, count, err := s.store.ListFlows(10, 0, flowType)
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@backend/internal/flow/mgt/store_test.go` around lines 156 - 162, Replace the
hardcoded "authentication" flow-type string in the test with the canonical enum
value so the test uses the real constant; update the two places where
s.mockDBClient.EXPECT().Query is called (the calls using queryCountFlowsWithType
and queryListFlowsWithType) and the ListFlows invocation to pass
string(common.FlowTypeAuthentication) instead of "authentication", and
add/ensure the test imports the package that defines FlowTypeAuthentication so
the symbol resolves; update only the test assertions/expectations to use this
canonical value.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@backend/dbscripts/thunderdb/postgres.sql`:
- Around line 309-317: Change the table to use a composite primary key of
(FLOW_ID, DEPLOYMENT_ID) instead of making FLOW_ID the standalone primary key:
remove the "PRIMARY KEY" annotation from FLOW_ID, remove the separate "UNIQUE
(FLOW_ID, DEPLOYMENT_ID)" constraint, and add a single "PRIMARY KEY (FLOW_ID,
DEPLOYMENT_ID)" definition; keep the other unique constraint "UNIQUE (HANDLE,
FLOW_TYPE, DEPLOYMENT_ID)" as-is and ensure DEPLOYMENT_ID column is present and
NOT NULL to support the new composite primary key.

In `@backend/dbscripts/thunderdb/sqlite.sql`:
- Around line 306-314: The FLOW table currently declares FLOW_ID as the
standalone PRIMARY KEY which enforces global uniqueness; change the primary key
to the composite (FLOW_ID, DEPLOYMENT_ID) to scope rows per deployment, remove
the redundant UNIQUE (FLOW_ID, DEPLOYMENT_ID) constraint, and keep or adjust
other UNIQUE constraints (e.g., UNIQUE (HANDLE, FLOW_TYPE, DEPLOYMENT_ID)) so
they still include DEPLOYMENT_ID; update any code or FK definitions that assume
a single-column primary key (references to FLOW_ID) to use the composite key
where appropriate.

---

Nitpick comments:
In `@backend/internal/flow/mgt/store_test.go`:
- Around line 156-162: Replace the hardcoded "authentication" flow-type string
in the test with the canonical enum value so the test uses the real constant;
update the two places where s.mockDBClient.EXPECT().Query is called (the calls
using queryCountFlowsWithType and queryListFlowsWithType) and the ListFlows
invocation to pass string(common.FlowTypeAuthentication) instead of
"authentication", and add/ensure the test imports the package that defines
FlowTypeAuthentication so the symbol resolves; update only the test
assertions/expectations to use this canonical value.

ℹ️ Review info

Configuration used: defaults

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between bc0ebe0 and 7cb9f0e.

📒 Files selected for processing (5)
  • backend/dbscripts/thunderdb/postgres.sql
  • backend/dbscripts/thunderdb/sqlite.sql
  • backend/internal/flow/mgt/store.go
  • backend/internal/flow/mgt/store_constants.go
  • backend/internal/flow/mgt/store_test.go

@codecov
Copy link

codecov bot commented Feb 26, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.41%. Comparing base (977c4a1) to head (81d3750).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1584      +/-   ##
==========================================
+ Coverage   91.35%   91.41%   +0.05%     
==========================================
  Files         692      692              
  Lines       46194    46155      -39     
==========================================
- Hits        42202    42194       -8     
+ Misses       2673     2659      -14     
+ Partials     1319     1302      -17     
Flag Coverage Δ
backend-integration-postgres 52.56% <66.66%> (+0.02%) ⬆️
backend-integration-sqlite 52.53% <66.66%> (+0.02%) ⬆️
backend-unit 82.84% <100.00%> (+0.19%) ⬆️
frontend-apps-develop-unit 97.55% <ø> (-0.01%) ⬇️
frontend-apps-gate-unit 99.76% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
backend/internal/flow/mgt/store_test.go (1)

716-785: ⚠️ Potential issue | 🟠 Major

Add success path tests for transactional methods to meet coverage requirements.

The error path tests are comprehensive, but success paths for CreateFlow (57.1%), UpdateFlow (22.7%), and RestoreFlowVersion (11.1%) must be added to meet the 80% coverage guideline for each method. The current overall package coverage is 86.0%, but these critical transactional methods require success path testing to reach the required threshold.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@backend/internal/flow/mgt/store_test.go` around lines 716 - 785, Add unit
tests covering the success paths for CreateFlow, UpdateFlow, and
RestoreFlowVersion: for each test obtain a mock transaction from
GetConfigDBClient().BeginTx(), set Exec expectations on the mockTx for the
relevant queries (e.g., the insert used in CreateFlow, the update used in
UpdateFlow, and the restore/version-update query used in RestoreFlowVersion) to
return expected results (no error), expect mockTx.Commit() to be called and no
Rollback, call the store method (CreateFlow/UpdateFlow/RestoreFlowVersion) and
assert no error and that the returned result matches the
created/updated/restored flow/version; mirror existing error-path tests’
structure (use mockTx from modelmock.NewTxInterfaceMock,
s.mockDBProvider.EXPECT(), s.mockDBClient.EXPECT().BeginTx().Return(mockTx,
nil), then Exec expectations and Commit().Return(nil)) so coverage for those
transactional methods reaches the success-case paths.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Outside diff comments:
In `@backend/internal/flow/mgt/store_test.go`:
- Around line 716-785: Add unit tests covering the success paths for CreateFlow,
UpdateFlow, and RestoreFlowVersion: for each test obtain a mock transaction from
GetConfigDBClient().BeginTx(), set Exec expectations on the mockTx for the
relevant queries (e.g., the insert used in CreateFlow, the update used in
UpdateFlow, and the restore/version-update query used in RestoreFlowVersion) to
return expected results (no error), expect mockTx.Commit() to be called and no
Rollback, call the store method (CreateFlow/UpdateFlow/RestoreFlowVersion) and
assert no error and that the returned result matches the
created/updated/restored flow/version; mirror existing error-path tests’
structure (use mockTx from modelmock.NewTxInterfaceMock,
s.mockDBProvider.EXPECT(), s.mockDBClient.EXPECT().BeginTx().Return(mockTx,
nil), then Exec expectations and Commit().Return(nil)) so coverage for those
transactional methods reaches the success-case paths.

ℹ️ Review info

Configuration used: defaults

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 7cb9f0e and fdbc343.

📒 Files selected for processing (5)
  • backend/dbscripts/thunderdb/postgres.sql
  • backend/dbscripts/thunderdb/sqlite.sql
  • backend/internal/flow/mgt/store.go
  • backend/internal/flow/mgt/store_constants.go
  • backend/internal/flow/mgt/store_test.go

@KaveeshaPiumini KaveeshaPiumini force-pushed the refactor branch 2 times, most recently from 746df2f to 1e6ebbf Compare February 26, 2026 13:09
Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@backend/internal/flow/mgt/store_test.go`:
- Around line 156-162: Replace the hard-coded lowercase type string
"authentication" in the test expectations and the call to s.store.ListFlows with
the canonical FlowType constant (use string(common.FlowTypeAuthentication)) so
the query expectations on mockDBClient (queryCountFlowsWithType and
queryListFlowsWithType) and the ListFlows invocation use the canonical value;
update the two places where "authentication" is passed to use
string(common.FlowTypeAuthentication) to avoid case-sensitivity regressions.

ℹ️ Review info

Configuration used: defaults

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 746df2f and 1e6ebbf.

📒 Files selected for processing (5)
  • backend/dbscripts/thunderdb/postgres.sql
  • backend/dbscripts/thunderdb/sqlite.sql
  • backend/internal/flow/mgt/store.go
  • backend/internal/flow/mgt/store_constants.go
  • backend/internal/flow/mgt/store_test.go

@KaveeshaPiumini KaveeshaPiumini force-pushed the refactor branch 4 times, most recently from 3efa7fe to 437bc2e Compare February 26, 2026 15:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

breaking change The feature/ improvement will alter the existing behaviour Type/Improvement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants