From 3b43f53e7901d075928ac32a0923dfaeed5384af Mon Sep 17 00:00:00 2001 From: Fabio Lima Date: Sat, 21 Mar 2026 16:40:13 -0700 Subject: [PATCH 1/9] chore(docs): Add plan for Vancouver City's water fountains full sync --- docs/plans/README.md | 1 + .../vancouver-water-fountain-sync/plan.md | 242 ++++++++++++++++++ .../vancouver-water-fountain-sync/tracker.md | 150 +++++++++++ 3 files changed, 393 insertions(+) create mode 100644 docs/plans/vancouver-water-fountain-sync/plan.md create mode 100644 docs/plans/vancouver-water-fountain-sync/tracker.md diff --git a/docs/plans/README.md b/docs/plans/README.md index b1434bb..fcc310e 100644 --- a/docs/plans/README.md +++ b/docs/plans/README.md @@ -12,3 +12,4 @@ Implementation plans for Linkvan API development. | [Rails 8.1 Upgrade](rails-81-upgrade/plan.md) | Complete | 22/22 (100%) | 2026-03-15 | | [RuboCop Remediation](rubocop-remediation/plan.md) | Complete | 64/64 (100%) | 2026-03-14 | | [Test Coverage Implementation](test-coverage-implementation/plan.md) | Complete | 24/24 (100%) | 2026-01-26 | +| [Vancouver Water Fountain Sync](vancouver-water-fountain-sync/plan.md) | Not Started | 0/10 (0%) | 2026-03-21 | diff --git a/docs/plans/vancouver-water-fountain-sync/plan.md b/docs/plans/vancouver-water-fountain-sync/plan.md new file mode 100644 index 0000000..a9ac08a --- /dev/null +++ b/docs/plans/vancouver-water-fountain-sync/plan.md @@ -0,0 +1,242 @@ +# Vancouver City Water Fountain Sync Enhancement + +## Status: PENDING + +## Created: 2026-03-21 + +## Goal + +Enhance the Vancouver City API water fountain sync feature with two new capabilities: +1. **Full Sync**: Remove facilities not present in the API (soft-delete with `sync_removed` reason) +2. **Purge All**: Add ability to remove all water fountains via admin action +3. **Undelete Support**: Sync should undelete and update facilities that were previously soft-deleted but now appear in the API + +## Current State + +- `External::VancouverCity::Syncer` fetches facilities from Vancouver Open Data API +- `External::VancouverCity::FacilitySyncer` processes each record (create/update by external_id or name match) +- Facilities with `external_id` are tracked as "external" facilities +- No mechanism to remove facilities not present in the API +- Discarded (soft-deleted) facilities are not considered during sync + +## Target State + +- **Full Sync**: After fetching all records, any external facility for that API not in the response will be soft-deleted with `discard_reason: "sync_removed"` (enabled by default) +- **Undelete**: Discarded facilities matching API records will be undeleted and updated +- **Purge All**: New admin action to soft-delete all water fountains for a given API +- **Enhanced Result**: `Syncer.call` result includes `created_count`, `updated_count`, `deleted_count` + +## Analysis Summary + +### Key Changes Required + +1. **FacilitySyncer** - Modify to use `Facility.with_discarded` queries and call `undiscard` before updates +2. **Syncer** - Add `full_sync:` parameter (default `true`), track synced IDs, discard missing facilities +3. **PurgeService** - New service to purge all facilities for an API +4. **Admin Tools Controller** - Add `purge_facilities` action (DELETE method) +5. **Routes** - Add DELETE route for purge action +6. **Views** - Add checkbox for full sync and "Purge All" button + +### Discard Reason + +`"sync_removed"` - Indicates facility was removed during API synchronization + +### Dependencies + +- Rails `discard` gem (already in use via `Discardable` module) +- `External::ApiHelper` for API key validation and mapping + +### Breaking Changes + +- None - Full sync is enabled by default, but can be disabled via `full_sync: false` for incremental sync + +--- + +## Priority System + +- **CRITICAL** - Must complete for success +- **HIGH** - Should complete for full functionality +- **MEDIUM** - Recommended for best UX +- **LOW** - Optional improvements + +--- + +## Implementation Stages + +### Stage 1: Full Sync with Deletion + Undelete Support + +**Focus:** Modify FacilitySyncer and Syncer to support full sync with deletion and undelete + +#### 1.1 Add Tests for Facility Undeletion +- **Priority:** CRITICAL +- **Type:** Test +- **Location:** `spec/services/external/vancouver_city/facility_syncer/undelete_facility_spec.rb` +- **Description:** Add tests for undeleting discarded facilities during sync +- **Tests:** + - Discarded facility with matching `external_id` → undeleted and updated + - Discarded facility with matching name (internal update) → undeleted and services added + - Verify `undiscard` is called before update operations + +#### 1.2 Update FacilitySyncer to Handle Discarded Facilities +- **Priority:** CRITICAL +- **Type:** Code Fix +- **Location:** `app/services/external/vancouver_city/facility_syncer.rb` +- **Description:** Modify queries to use `with_discarded` and add undiscard logic +- **Changes:** + - Line 32: `Facility.find_by(external_id: ...)` → `Facility.with_discarded.find_by(external_id: ...)` + - Line 36: Name-match query also use `with_discarded` + - Add `facility.undiscard` before `update_external_facility` and `update_internal_facility` + +#### 1.3 Add Tests for Full Sync Deletion +- **Priority:** CRITICAL +- **Type:** Test +- **Location:** `spec/services/external/vancouver_city/syncer_spec.rb` +- **Description:** Add tests for `full_sync` option +- **Tests:** + - `full_sync: true` (default) → orphaned facilities discarded with `"sync_removed"` + - `full_sync: false` → no deletion (facilities kept even if missing from API) + - Result includes `created_count`, `updated_count`, `deleted_count` + +#### 1.4 Implement Full Sync Deletion Logic +- **Priority:** CRITICAL +- **Type:** Code Fix +- **Location:** `app/services/external/vancouver_city/syncer.rb` +- **Description:** Add `full_sync` parameter and deletion logic +- **Changes:** + - Add `full_sync:` boolean parameter (default `true`) to `initialize` + - Track fetched `external_id`s during `call` + - After processing all records, if `full_sync: true`: discard facilities not in response + +#### 1.5 Enhance Result with Counts +- **Priority:** HIGH +- **Type:** Code Fix +- **Location:** `app/services/external/vancouver_city/syncer.rb` +- **Description:** Add operation counts to result data +- **Changes:** Result data includes: + ```ruby + { + facilities: facilities, + total_count: facilities.size, + created_count: , + updated_count: , + deleted_count: , + api_key: api_key + } + ``` + +--- + +### Stage 2: Purge All Water Fountains + +**Focus:** Add ability to remove all water fountains for an API + +#### 2.1 Add Tests for Purge Service +- **Priority:** CRITICAL +- **Type:** Test +- **Location:** `spec/services/external/vancouver_city/purge_service_spec.rb` +- **Description:** Add tests for purge service +- **Tests:** + - Purges all external facilities for `api_key` + - Discards with `discard_reason: "sync_removed"` + - Returns `discarded_count` + - Validates `api_key` is supported + +#### 2.2 Implement Purge Service +- **Priority:** CRITICAL +- **Type:** Code Fix +- **Location:** `app/services/external/vancouver_city/purge_service.rb` +- **Description:** Create new service to purge all facilities for an API +- **Implementation:** Find all external facilities for the service, discard each with `sync_removed` reason + +#### 2.3 Add Tests for Purge Controller Action +- **Priority:** CRITICAL +- **Type:** Test +- **Location:** `spec/controllers/admin/tools_controller_spec.rb` +- **Description:** Add tests for purge action +- **Tests:** + - `DELETE /admin/tools/purge_facilities?api=drinking-fountains` + - Admin only; non-admins redirect with access denied + - Success redirects with notice showing count + - Invalid `api_key` returns alert + +#### 2.4 Implement Purge Action + Route +- **Priority:** CRITICAL +- **Type:** Code Fix +- **Location:** `app/controllers/admin/tools_controller.rb`, `config/routes.rb` +- **Description:** Add purge_facilities action and DELETE route +- **Changes:** + - Add `purge_facilities` action (DELETE method) + - Add route: `delete :purge_facilities, to: 'admin/tools#purge_facilities'` + +--- + +### Stage 3: Update Admin Tools View + +#### 3.1 Update Admin Tools View +- **Priority:** HIGH +- **Type:** Code Fix +- **Location:** `app/views/admin/tools/index.html.*` +- **Description:** Add UI for full sync and purge options +- **Changes:** + - Remove checkbox (full sync is now the default behavior) + - Add "Purge All" button with confirmation dialog → `DELETE /admin/tools/purge_facilities?api=drinking-fountains` + +--- + +## Quality Checks + +### Stage 1 Completion Criteria +- [ ] Undeletion tests pass (1.1) +- [ ] FacilitySyncer updated with undelete logic (1.2) +- [ ] Deletion tests pass (1.3) +- [ ] Full sync deletion logic implemented (1.4) +- [ ] Result enhanced with counts (1.5) +- [ ] Run `bin/rspec spec/services/external/vancouver_city/` + +### Stage 2 Completion Criteria +- [ ] Purge service tests pass (2.1) +- [ ] Purge service implemented (2.2) +- [ ] Purge controller tests pass (2.3) +- [ ] Purge action and route implemented (2.4) +- [ ] Run `bin/rspec spec/controllers/admin/tools_controller_spec.rb` + +### Stage 3 Completion Criteria +- [ ] View updated with checkbox (3.1) +- [ ] Manual test: Verify UI elements work correctly + +### Overall Completion Criteria +- [ ] All tests pass +- [ ] `bin/rubocop` passes +- [ ] Manual verification of full sync and purge features + +--- + +## Rollback Plan + +If issues occur: +1. Revert `app/services/external/vancouver_city/syncer.rb` - removes `full_sync` and deletion logic +2. Revert `app/services/external/vancouver_city/facility_syncer.rb` - removes `with_discarded` and undelete logic +3. Revert `app/controllers/admin/tools_controller.rb` - removes `purge_facilities` action +4. Revert `config/routes.rb` - removes purge route +5. Delete new files: `purge_service.rb`, `undelete_facility_spec.rb` + +--- + +## Estimated Time + +| Stage | Tasks | Time | +|-------|-------|------| +| 1 | 5 | 45 min | +| 2 | 4 | 30 min | +| 3 | 1 | 10 min | +| Total | 10 | ~85 min | + +--- + +## Related Documentation + +- [Vancouver City API Syncer](../../app/services/external/vancouver_city/syncer.rb) +- [Vancouver City FacilitySyncer](../../app/services/external/vancouver_city/facility_syncer.rb) +- [AGENTS.md](../../AGENTS.md) +- [Rails Migrations Skill](../../.opencode/skills/rails-migrations/SKILL.md) +- [RSpec Testing Skill](../../.opencode/skills/rspec-testing/SKILL.md) diff --git a/docs/plans/vancouver-water-fountain-sync/tracker.md b/docs/plans/vancouver-water-fountain-sync/tracker.md new file mode 100644 index 0000000..1a221ff --- /dev/null +++ b/docs/plans/vancouver-water-fountain-sync/tracker.md @@ -0,0 +1,150 @@ +# Vancouver City Water Fountain Sync Enhancement Tracker + +## Plan Reference + +[plan.md](plan.md) + +--- + +## Created: 2026-03-21 +## Last Updated: 2026-03-21 + +--- + +## Summary + +| Priority | Total | Not Started | In Progress | Completed | Blocked | +|----------|-------|-------------|-------------|-----------|---------| +| CRITICAL | 8 | 8 | 0 | 0 | 0 | +| HIGH | 2 | 2 | 0 | 0 | 0 | +| **TOTAL**| **10**| **10** | **0** | **0** | **0** | + +--- + +## Stage 1: Full Sync with Deletion + Undelete Support + +### Item Tables + +#### 1.1 - Add Tests for Facility Undeletion + +| ID | Priority | Status | File | Notes | +|----|----------|--------|------|-------| +| 1.1 | CRITICAL | ⬜ Not Started | spec/.../facility_syncer/undelete_facility_spec.rb | New test file | + +#### 1.2 - Update FacilitySyncer to Handle Discarded Facilities + +| ID | Priority | Status | File | Notes | +|----|----------|--------|------|-------| +| 1.2 | CRITICAL | ⬜ Not Started | app/.../facility_syncer.rb | Use with_discarded, add undiscard | + +#### 1.3 - Add Tests for Full Sync Deletion + +| ID | Priority | Status | File | Notes | +|----|----------|--------|------|-------| +| 1.3 | CRITICAL | ⬜ Not Started | spec/.../syncer_spec.rb | Tests for full_sync option (default true) | + +#### 1.4 - Implement Full Sync Deletion Logic + +| ID | Priority | Status | File | Notes | +|----|----------|--------|------|-------| +| 1.4 | CRITICAL | ⬜ Not Started | app/.../syncer.rb | Add full_sync param (default true) + deletion | + +#### 1.5 - Enhance Result with Counts + +| ID | Priority | Status | File | Notes | +|----|----------|--------|------|-------| +| 1.5 | HIGH | ⬜ Not Started | app/.../syncer.rb | Add created/updated/deleted counts | + +--- + +## Stage 2: Purge All Water Fountains + +### Item Tables + +#### 2.1 - Add Tests for Purge Service + +| ID | Priority | Status | File | Notes | +|----|----------|--------|------|-------| +| 2.1 | CRITICAL | ⬜ Not Started | spec/.../purge_service_spec.rb | New test file | + +#### 2.2 - Implement Purge Service + +| ID | Priority | Status | File | Notes | +|----|----------|--------|------|-------| +| 2.2 | CRITICAL | ⬜ Not Started | app/.../purge_service.rb | New service file | + +#### 2.3 - Add Tests for Purge Controller Action + +| ID | Priority | Status | File | Notes | +|----|----------|--------|------|-------| +| 2.3 | CRITICAL | ⬜ Not Started | spec/controllers/admin/tools_controller_spec.rb | Add purge tests | + +#### 2.4 - Implement Purge Action + Route + +| ID | Priority | Status | File | Notes | +|----|----------|--------|------|-------| +| 2.4 | CRITICAL | ⬜ Not Started | app/.../tools_controller.rb, config/routes.rb | Add action + route | + +--- + +## Stage 3: Update Admin Tools View + +### Item Tables + +#### 3.1 - Update Admin Tools View + +| ID | Priority | Status | File | Notes | +|----|----------|--------|------|-------| +| 3.1 | HIGH | ⬜ Not Started | app/views/admin/tools/index.html.* | Add "Purge All" button (full sync is default) | + +--- + +## Dependencies + +- Stage 1 must complete before Stage 2 +- Stage 1.2 must complete before 1.4 (FacilitySyncer changes needed for Syncer) + +### Blockers + +None identified at this time. + +--- + +## Progress Tracking + +``` +Stage 1 (CRITICAL): ░░░░░░░░░░░░░░░░░░░░ 0/5 items (0%) +Stage 2 (CRITICAL): ░░░░░░░░░░░░░░░░░░░░ 0/4 items (0%) +Stage 3 (HIGH): ░░░░░░░░░░░░░░░░░░░░ 0/1 items (0%) +Overall: ░░░░░░░░░░░░░░░░░░░░ 0/10 items (0%) +``` + +--- + +## Status Legend + +| Icon | Status | +|------|--------| +| ⬜ | Not Started | +| 🔄 | In Progress | +| ✅ | Completed | +| ⏸️ | On Hold | +| 🚫 | Blocked | + +--- + +## Change Log + +| Date | Change | Author | +|------|--------|--------| +| 2026-03-21 | Initial plan creation | Assistant | + +--- + +## Notes + +- Discard reason for removed facilities: `sync_removed` +- `full_sync: true` (default) - soft-deletes missing facilities; use `full_sync: false` for incremental sync +- Purge action uses DELETE HTTP method +- Confirmation dialog required before purge +- Since discard is used, operation is soft-delete (non-destructive, reversible) From 00ab4a66d84f404f24a4bbc9b9cfb48886261caa Mon Sep 17 00:00:00 2001 From: Fabio Lima Date: Sat, 11 Apr 2026 09:55:23 -0700 Subject: [PATCH 2/9] chore(ai): Update subagents models to MiniMax-M2.7 --- .opencode/agents/rails-code-auditor.md | 2 +- .opencode/agents/rails-migration-manager.md | 2 +- .opencode/agents/rails-refactor.md | 2 +- .opencode/agents/rails-resource-builder.md | 2 +- .opencode/agents/rails-test-runner.md | 2 +- .opencode/agents/ruby-gem-updater.md | 2 +- 6 files changed, 6 insertions(+), 6 deletions(-) diff --git a/.opencode/agents/rails-code-auditor.md b/.opencode/agents/rails-code-auditor.md index 8e40ed1..cc56bcb 100644 --- a/.opencode/agents/rails-code-auditor.md +++ b/.opencode/agents/rails-code-auditor.md @@ -1,7 +1,7 @@ --- description: Review code for quality and Rails conventions (report + suggest on request) mode: subagent -model: minimax-coding-plan/MiniMax-M2.5 +model: minimax-coding-plan/MiniMax-M2.7 permission: skill: "rails-code-quality": "allow" diff --git a/.opencode/agents/rails-migration-manager.md b/.opencode/agents/rails-migration-manager.md index 86e1c41..f84e53f 100644 --- a/.opencode/agents/rails-migration-manager.md +++ b/.opencode/agents/rails-migration-manager.md @@ -1,7 +1,7 @@ --- description: Manage Rails migrations - create, run, rollback, and troubleshoot mode: subagent -model: minimax-coding-plan/MiniMax-M2.5 +model: minimax-coding-plan/MiniMax-M2.7 permission: skill: "rails-migrations": "allow" diff --git a/.opencode/agents/rails-refactor.md b/.opencode/agents/rails-refactor.md index 5e439bd..ef9f840 100644 --- a/.opencode/agents/rails-refactor.md +++ b/.opencode/agents/rails-refactor.md @@ -1,7 +1,7 @@ --- description: Refactor code following Rails and project conventions mode: subagent -model: minimax-coding-plan/MiniMax-M2.5 +model: minimax-coding-plan/MiniMax-M2.7 permission: skill: "rails-code-quality": "allow" diff --git a/.opencode/agents/rails-resource-builder.md b/.opencode/agents/rails-resource-builder.md index 7844740..dea9129 100644 --- a/.opencode/agents/rails-resource-builder.md +++ b/.opencode/agents/rails-resource-builder.md @@ -1,7 +1,7 @@ --- description: Generate complete Rails resources (models, controllers, routes, tests) mode: subagent -model: minimax-coding-plan/MiniMax-M2.5 +model: minimax-coding-plan/MiniMax-M2.7 permission: skill: "rails-models": "allow" diff --git a/.opencode/agents/rails-test-runner.md b/.opencode/agents/rails-test-runner.md index 9f9819b..c590696 100644 --- a/.opencode/agents/rails-test-runner.md +++ b/.opencode/agents/rails-test-runner.md @@ -1,7 +1,7 @@ --- description: Execute tests and report results only mode: subagent -model: minimax-coding-plan/MiniMax-M2.5 +model: minimax-coding-plan/MiniMax-M2.7 permission: skill: "rspec-testing": "allow" diff --git a/.opencode/agents/ruby-gem-updater.md b/.opencode/agents/ruby-gem-updater.md index 1cb9680..b5a0cc8 100644 --- a/.opencode/agents/ruby-gem-updater.md +++ b/.opencode/agents/ruby-gem-updater.md @@ -1,7 +1,7 @@ --- description: Execute Ruby gem updates with version checking, breaking change analysis, and testing mode: subagent -model: minimax-coding-plan/MiniMax-M2.5 +model: minimax-coding-plan/MiniMax-M2.7 permission: skill: "ruby-gem-update": "allow" From 947193a8d94b2facf7895ddac2eedf7cdeb468f3 Mon Sep 17 00:00:00 2001 From: Fabio Lima Date: Sat, 11 Apr 2026 10:49:29 -0700 Subject: [PATCH 3/9] feat(vancouver-city): enhance water fountain sync with soft-delete and purge support - Add soft-delete for facilities absent from API response (enabled by default) - Sync now undeletes and updates previously soft-deleted facilities - New admin purge action to remove all water fountains for a given API - Enhanced Syncer.call result with created_count, updated_count, deleted_count --- app/controllers/admin/tools_controller.rb | 28 +- app/models/facility.rb | 12 +- app/services/application_service.rb | 8 + .../vancouver_city/facility_syncer.rb | 12 +- .../external/vancouver_city/purge_service.rb | 24 ++ .../external/vancouver_city/syncer.rb | 79 ++++- app/views/admin/tools/index.html.erb | 34 ++- config/routes.rb | 1 + .../vancouver-water-fountain-sync/plan.md | 2 +- .../vancouver-water-fountain-sync/tracker.md | 37 +-- .../admin/tools_controller_spec.rb | 146 ++++++++++ spec/factories/facilities.rb | 7 + spec/models/facility_spec.rb | 10 +- .../facility_syncer/error_handling_spec.rb | 8 +- .../external_update_operation_spec.rb | 16 +- .../internal_update_operation_spec.rb | 48 ++-- .../facility_syncer/undelete_facility_spec.rb | 272 ++++++++++++++++++ .../vancouver_city/purge_service_spec.rb | 85 ++++++ .../external/vancouver_city/syncer_spec.rb | 189 ++++++++++++ 19 files changed, 954 insertions(+), 64 deletions(-) create mode 100644 app/services/external/vancouver_city/purge_service.rb create mode 100644 spec/controllers/admin/tools_controller_spec.rb create mode 100644 spec/services/external/vancouver_city/facility_syncer/undelete_facility_spec.rb create mode 100644 spec/services/external/vancouver_city/purge_service_spec.rb diff --git a/app/controllers/admin/tools_controller.rb b/app/controllers/admin/tools_controller.rb index 0448a6e..3b3f1d3 100644 --- a/app/controllers/admin/tools_controller.rb +++ b/app/controllers/admin/tools_controller.rb @@ -16,18 +16,40 @@ def import_facilities result = External::VancouverCity::Syncer.call( api_key: api_key, - api_client: External::VancouverCity.default_client + api_client: External::VancouverCity.default_client, + full_sync: true ) if result.success? - total_count = result.data[:total_count] || 0 - redirect_to admin_facilities_path(service: "water_fountain"), notice: "#{total_count} Facilities imported successfully from #{External::ApiHelper.api_name(api_key)}." + created = result.data[:created_count] || 0 + updated = result.data[:updated_count] || 0 + deleted = result.data[:deleted_count] || 0 + redirect_to admin_facilities_path(service: "water_fountain"), notice: "Sync complete: #{created} created, #{updated} updated, #{deleted} removed from #{External::ApiHelper.api_name(api_key)}." else error_messages = result.errors.join(", ") redirect_to admin_tools_path, alert: "Failed to import facilities: #{error_messages}" end end + def purge_facilities + api_key = params[:api] + + unless External::ApiHelper.supported_api?(api_key) + redirect_to admin_tools_path, alert: "Invalid API selected. Please choose from the supported APIs." + return + end + + result = External::VancouverCity::PurgeService.call(api_key: api_key) + + if result.success? + discarded_count = result.data[:discarded_count] || 0 + redirect_to admin_facilities_path(service: "water_fountain"), notice: "#{discarded_count} facilities purged from #{External::ApiHelper.api_name(api_key)}." + else + error_messages = result.errors.join(", ") + redirect_to admin_tools_path, alert: "Failed to purge facilities: #{error_messages}" + end + end + # Helper method for the view helper_method :api_options_for_select diff --git a/app/models/facility.rb b/app/models/facility.rb index 450cc5e..e510456 100644 --- a/app/models/facility.rb +++ b/app/models/facility.rb @@ -17,10 +17,10 @@ class Facility < ApplicationRecord has_many :time_slots, through: :schedules enum :discard_reason, { - none: nil, closed: "closed", - duplicated: "duplicated" - }, prefix: true, default: :none + duplicated: "duplicated", + sync_removed: "sync_removed" + }, prefix: true, default: nil validates :name, presence: true validate :validate_website @@ -43,6 +43,10 @@ class Facility < ApplicationRecord scope :external, -> { where.not(external_id: nil) } scope :not_external, -> { where(external_id: nil) } + def discard_reason_none? + discard_reason.nil? + end + def managed_by?(user_or_user_id) return false if user_or_user_id.blank? @@ -157,7 +161,7 @@ def clean_data end # handles discard - self.discard_reason = :none if undiscarded? + self.discard_reason = nil if undiscarded? end def distance(to_coord: nil, to_lat: nil, to_long: nil, to_facility: nil) diff --git a/app/services/application_service.rb b/app/services/application_service.rb index cb9d386..b6bc76f 100644 --- a/app/services/application_service.rb +++ b/app/services/application_service.rb @@ -31,6 +31,14 @@ def invalid? !valid? end + def success(data = {}) + Result.new(data: data, errors: []) + end + + def failure(errors) + Result.new(data: nil, errors: Array(errors)) + end + private def errors diff --git a/app/services/external/vancouver_city/facility_syncer.rb b/app/services/external/vancouver_city/facility_syncer.rb index 1ffbd83..526d7a4 100644 --- a/app/services/external/vancouver_city/facility_syncer.rb +++ b/app/services/external/vancouver_city/facility_syncer.rb @@ -29,11 +29,13 @@ def call end built_facility = builder_result.data[:facility] - existing_facility = Facility.find_by(external_id: built_facility.external_id) + existing_facility = Facility.with_discarded + .find_by(external_id: built_facility.external_id) # If no external_id match, look for name match but prefer internal facilities if existing_facility.blank? - existing_facility = Facility.where(name: built_facility.name) + existing_facility = Facility.with_discarded + .where(name: built_facility.name) .order(Arel.sql("external_id IS NULL DESC, external_id")) .first end @@ -84,12 +86,15 @@ def call private def update_internal_facility(internal_facility, built_facility) + internal_facility.undiscard if internal_facility.discarded? + add_missing_services(internal_facility, built_facility) end def update_external_facility(external_facility, built_facility) - add_missing_services(external_facility, built_facility) + external_facility.undiscard if external_facility.discarded? + add_missing_services(external_facility, built_facility) external_facility.update!(built_facility.attributes.slice("name", "address", "lat", "long", "verified")) end @@ -97,6 +102,7 @@ def add_missing_services(existing_facility, built_facility) built_services = built_facility.facility_services.map(&:service).uniq existing_services = existing_facility.facility_services.map(&:service).uniq new_services = built_services - existing_services + new_services.each do |service| existing_facility.facility_services.create!(service: service) end diff --git a/app/services/external/vancouver_city/purge_service.rb b/app/services/external/vancouver_city/purge_service.rb new file mode 100644 index 0000000..6e29df8 --- /dev/null +++ b/app/services/external/vancouver_city/purge_service.rb @@ -0,0 +1,24 @@ +# frozen_string_literal: true + +class External::VancouverCity::PurgeService < ApplicationService + attr_reader :api_key + + def initialize(api_key:) + super() + @api_key = api_key + end + + def call + return failure(["Unsupported API: #{api_key}"]) unless External::ApiHelper.supported_api?(api_key) + + discarded_count = 0 + + Facility.external.kept.find_each do |facility| + facility.discard_reason = :sync_removed + facility.discard! + discarded_count += 1 + end + + success({ discarded_count: discarded_count }) + end +end diff --git a/app/services/external/vancouver_city/syncer.rb b/app/services/external/vancouver_city/syncer.rb index b8591fa..861d7c9 100644 --- a/app/services/external/vancouver_city/syncer.rb +++ b/app/services/external/vancouver_city/syncer.rb @@ -3,17 +3,19 @@ # Service for syncing facility data from Vancouver City Open Data API # Inherits from ApplicationService and handles pagination to fetch all facilities class External::VancouverCity::Syncer < ApplicationService - attr_reader :api_key, :api_client + attr_reader :api_key, :api_client, :full_sync PAGE_SIZE = 50 # Maximum records per request allowed by the API # Initialize the syncer with required parameters # @param api_key [String] One of the supported API keys from External::ApiHelper # @param api_client [VancouverApiClient] The API client instance - def initialize(api_key:, api_client:) + # @param full_sync [Boolean] Whether to perform a full sync (discard missing facilities) + def initialize(api_key:, api_client:, full_sync: true) super() @api_key = api_key @api_client = api_client + @full_sync = full_sync end # Main method that performs the sync operation @@ -22,6 +24,9 @@ def call return Result.new(data: nil, errors: errors) if invalid? facilities = [] + synced_external_ids = [] + created_count = 0 + updated_count = 0 offset = 0 loop do @@ -34,8 +39,17 @@ def call break if records.empty? # Process each record and build Facility objects - batch_facilities = process_records(records) - facilities.concat(batch_facilities) + batch_results = process_records_with_operations(records) + batch_results.each do |result| + facilities << result.facility + synced_external_ids << result.facility.external_id if result.facility.respond_to?(:external_id) + case result.operation + when :create + created_count += 1 + when :external_update, :internal_update + updated_count += 1 + end + end # If we got fewer records than the limit, we've reached the end break if records.size < PAGE_SIZE @@ -50,12 +64,17 @@ def call end end + discarded_count = discard_missing_facilities(synced_external_ids) + Rails.logger.info "Successfully processed #{facilities.size} facilities from #{api_key} API" Result.new( data: { facilities: facilities, total_count: facilities.size, + created_count: created_count, + updated_count: updated_count, + deleted_count: discarded_count, api_key: api_key }, errors: errors @@ -98,4 +117,56 @@ def process_records(records) facilities end + + # Process API records and return ResultData objects with operations + # @param records [Array] Array of API response records + # @return [Array] Array of result data objects + def process_records_with_operations(records) + results = [] + + records.each do |record| + syncer_result = External::VancouverCity::FacilitySyncer.call(record: record, api_key: api_key) + + if syncer_result.success? + data = syncer_result.data + # Support both ResultData objects and legacy hash format + results << if data.respond_to?(:operation) + data + else + # Legacy hash format: { facility: ... } + External::VancouverCity::FacilitySyncer::ResultData.new( + operation: nil, + facility: data[:facility] + ) + end + else + add_errors(syncer_result.errors) + end + end + + results + end + + # Discard facilities that were not in the API response (full sync only) + # @param synced_external_ids [Array] Array of external_ids that were in the response + # @return [Integer] Number of facilities discarded + def discard_missing_facilities(synced_external_ids) + return 0 unless full_sync + + # Only discard facilities that: + # 1. Are currently kept (not already discarded with deleted_at set) + # 2. Have not been previously marked as sync_removed + missing_facilities = Facility.external.kept + .where.not(external_id: synced_external_ids) + .where("discard_reason IS NULL OR discard_reason != ?", "sync_removed") + + count = 0 + missing_facilities.find_each do |facility| + facility.discard_reason = :sync_removed + facility.discard! + count += 1 + end + + count + end end diff --git a/app/views/admin/tools/index.html.erb b/app/views/admin/tools/index.html.erb index 72f3707..aa97257 100644 --- a/app/views/admin/tools/index.html.erb +++ b/app/views/admin/tools/index.html.erb @@ -29,12 +29,44 @@ +
+
+ <%= form.submit "Import Facilities", class: "button is-primary", id: "import-button", data: { disable_with: "Importing..." } %> +
+
+ +
+

Note: Import performs a full sync, which removes any facilities that no longer exist in the API.

+
+ <% end %> + + +
+ <%= render Shared::CardComponent.new(title: "Purge Facilities") do |card| %> +
+

Remove all water fountains from the database. This action cannot be undone.

+ + <%= form_with url: purge_facilities_admin_tools_path, method: :delete, class: "form", id: "purge-form" do |form| %>
+ <%= form.label :api, "API Endpoint", class: "label" %>
- <%= form.submit "Import Facilities", class: "button is-primary", id: "import-button", data: { disable_with: "Importing..." } %> +
+ <%= form.select :api, + options_for_select(api_options_for_select), + { include_blank: 'Select an API...' }, + { class: "select", required: true } %> +
+
+
+ +
+
+ <%= form.submit "Purge All Facilities", class: "button is-danger", id: "purge-button", data: { confirm: "Are you sure you want to remove ALL facilities for the selected API? This action cannot be undone.", disable_with: "Purging..." } %>
<% end %> +
+ <% end %>