From ae69d579c0817912ceae2b328aa872dd7402081e Mon Sep 17 00:00:00 2001 From: zewelor Date: Sat, 21 Mar 2026 15:24:36 +0000 Subject: [PATCH 1/7] Add Google Sheets, Gmail, and OAuth2 plans - Add planning docs for Gmail output, Google Sheets client, and OAuth2 - Provide API sketches, ClientProxy integration, env/vault naming, flows - Recommend gems and describe modes, error handling, and usage examples - Add a one-line todo about implementing ActiveJob Continuation for workflows --- docs/todo/plan-gmail-output.md | 228 +++++++++++++++++++++++++ docs/todo/plan-google-oauth2.md | 154 +++++++++++++++++ docs/todo/plan-google-sheets-client.md | 169 ++++++++++++++++++ 3 files changed, 551 insertions(+) create mode 100644 docs/todo/plan-gmail-output.md create mode 100644 docs/todo/plan-google-oauth2.md create mode 100644 docs/todo/plan-google-sheets-client.md diff --git a/docs/todo/plan-gmail-output.md b/docs/todo/plan-gmail-output.md new file mode 100644 index 00000000..180944c0 --- /dev/null +++ b/docs/todo/plan-gmail-output.md @@ -0,0 +1,228 @@ +# Plan: Gmail Output + +## Overview + +R3x output for sending emails via Gmail API with test mode for safe development. +Follows the `R3x::Outputs::Discord` pattern. + +## Files + +1. `Gemfile` — add `mail` gem (if not present) +2. `app/lib/r3x/outputs/gmail.rb` — output implementation +3. `lib/r3x/workflow/context.rb` — add to ClientProxy + +--- + +## 1. Gemfile + +Check if `mail` gem is present, if not add: + +```ruby +gem "mail" # For RFC 2822 email building +``` + +--- + +## 2. `app/lib/r3x/outputs/gmail.rb` + +### Class: `R3x::Outputs::Gmail` + +```ruby +module R3x + module Outputs + class Gmail + include R3x::Concerns::Logger + + def initialize(credentials:, mode: nil) + @credentials = credentials + @mode = mode || ENV.fetch("R3X_GMAIL_MODE", "test") + end + + def deliver(to:, subject:, body:) + case mode + when "real" + deliver_real(to: to, subject: subject, body: body) + when "test" + deliver_test(to: to, subject: subject, body: body) + else + raise ArgumentError, "Unsupported Gmail mode: #{mode}. Supported: real, test" + end + end + + private + + attr_reader :credentials, :mode + + def deliver_real(to:, subject:, body:) + service = Google::Apis::GmailV1::GmailService.new + service.authorization = R3x::Client::GoogleAuth.from_json( + credentials, + scope: Google::Apis::GmailV1::AUTH_GMAIL_SEND + ) + + mail = Mail.new do + to to + subject subject + body body + end + + message = Google::Apis::GmailV1::Message.new( + raw: Base64.urlsafe_encode64(mail.to_s) + ) + + result = service.send_user_message("me", message) + {"mode" => "real", "message_id" => result.id} + end + + def deliver_test(to:, subject:, body:) + logger.info("Gmail [TEST] to=#{to} subject=#{subject}\n#{body}") + {"mode" => "test"} + end + end + end +end +``` + +### Public API + +| Method | Returns | Description | +|--------|---------|-------------| +| `deliver(to:, subject:, body:)` | `Hash` | Send email (real mode) or log (test mode) | + +### Mode behavior + +| Mode | Env var value | Behavior | +|------|--------------|----------| +| `test` | default | Log to Rails logger, no API call | +| `real` | `"real"` | Send via Gmail API | + +### Error handling + +- `Google::Apis::ClientError` (400) — invalid recipient +- `Google::Apis::ClientError` (403) — quota exceeded or permission denied +- `Signet::AuthorizationError` — refresh token expired + +--- + +## 3. ClientProxy addition + +File: `lib/r3x/workflow/context.rb` + +Add inside `ClientProxy` class: + +```ruby +def gmail(credentials_env:, mode: nil) + credentials = fetch_google_credentials(credentials_env) + R3x::Outputs::Gmail.new(credentials: credentials, mode: mode) +end +``` + +The `fetch_google_credentials` private method is shared with `google_sheets` (see plan-google-sheets-client.md). + +--- + +## Env var + +```bash +R3X_GMAIL_MODE=test # or "real" for production +``` + +Can also be overridden per-call: +```ruby +ctx.client.gmail(credentials_env: "...", mode: "real").deliver(...) +``` + +--- + +## Usage in workflow + +```ruby +ctx.client.gmail( + credentials_env: "GOOGLE_CREDENTIALS_PXOPULSE" +).deliver( + to: "pxopulse@gmail.com", + subject: "Weekly pulse", + body: formatted_content +) + +# Test mode returns: +# {"mode" => "test"} + +# Real mode returns: +# {"mode" => "real", "message_id" => "18a3f..."} +``` + +--- + +## PxoWeekly email format + +Based on n8n workflow output: + +``` +🇬🇧🎉 This week in Porto Santo + +🌅 #Sunsessions with M da Silva +📍 Foot On Water Restaurant & Beach Bar +📅 17 de agosto, 16h30–20h30 + +🎤 Concert at Praça +📍 Town Square +📅 18 de agosto, 21h00 + +💬 More details on pxopulse.com + + +============== + +🇵🇹🎉 Esta semana em Porto Santo + +🌅 #Sunsessions com M da Silva +📍 Foot On Water Restaurant & Beach Bar +📅 17 de agosto, 16h30–20h30 + +🎤 Concerto na Praça +📍 Praça do Povo +📅 18 de agosto, 21h00 + +💬 Mais detalhes em pxopulse.com +``` + +--- + +## Comparison with Discord Output + +| Aspect | `Outputs::Discord` | `Outputs::Gmail` | +|--------|-------------------|------------------| +| Mode env var | `R3X_DISCORD_MODE` | `R3X_GMAIL_MODE` | +| Test output | Logs to Rails logger | Logs to Rails logger | +| Auth | Webhook URL | OAuth2 credentials | +| Content | Plain text | RFC 2822 email | +| Method | `deliver(content:)` | `deliver(to:, subject:, body:)` | + +--- + +## Gmail API limits + +- Personal account: ~100 emails/day +- Google Workspace: ~1500 emails/day +- OAuth app must be in "Testing" or "Published" status + +--- + +## Dependencies + +```ruby +gem "google-apis-gmail_v1" # Already in Gemfile +gem "mail" # Check/add for email building +gem "googleauth" # Already in Gemfile +``` + +--- + +## Related files + +- `app/lib/r3x/client/google_auth.rb` — shared OAuth2 module (plan: plan-google-oauth2.md) +- `app/lib/r3x/outputs/discord.rb` — similar output pattern +- `lib/r3x/workflow/context.rb` — ClientProxy integration +- `docs/todo/plan-google-oauth2.md` — OAuth2 setup +- `docs/todo/plan-google-sheets-client.md` — Sheets client diff --git a/docs/todo/plan-google-oauth2.md b/docs/todo/plan-google-oauth2.md new file mode 100644 index 00000000..93d57eb6 --- /dev/null +++ b/docs/todo/plan-google-oauth2.md @@ -0,0 +1,154 @@ +# Plan: Google OAuth2 Helper + +## Overview + +Interactive CLI helper for obtaining Google refresh tokens and shared OAuth2 credential module +for per-project Google API credentials stored in Vault. + +## Files + +1. `app/lib/r3x/client/google_auth.rb` — shared credential builder +2. `bin/google-oauth` — interactive CLI helper + +--- + +## 1. `app/lib/r3x/client/google_auth.rb` + +Shared module for building OAuth2 credentials from JSON. + +```ruby +module R3x::Client::GoogleAuth + SCOPE_ALIASES = { + sheets: Google::Apis::SheetsV4::AUTH_SPREADSHEETS_READONLY, + gmail: Google::Apis::GmailV1::AUTH_GMAIL_SEND + }.freeze + + def self.from_json(parsed_json, scope:) + Signet::OAuth2::Client.new( + client_id: parsed_json.fetch("client_id"), + client_secret: parsed_json.fetch("client_secret"), + refresh_token: parsed_json.fetch("refresh_token"), + token_credential_uri: "https://oauth2.googleapis.com/token", + scope: Array(scope) + ).tap(&:fetch_access_token!) + end +end +``` + +**Error handling:** +- `KeyError` if required fields missing from JSON +- Propagate `Signet::AuthorizationError` on invalid refresh token + +--- + +## 2. `bin/google-oauth` + +```bash +#!/usr/bin/env ruby +require_relative "../config/environment" +require "optparse" +``` + +### Commands + +| Command | Description | +|---------|-------------| +| `authorize --project PROJECT --scopes sheets,gmail` | Start OAuth2 flow | +| `status --project PROJECT` | Check credential status | + +### Pre-setup (manual) + +1. Google Cloud Console → create project +2. Enable APIs: Sheets API + Gmail API +3. OAuth consent screen → External → Add test users +4. Credentials → Create OAuth client ID → Desktop app +5. Extract `client_id` and `client_secret` +6. Store in Vault: `GOOGLE_CLIENT_ID_`, `GOOGLE_CLIENT_SECRET_` + +### `authorize` flow + +1. Read `GOOGLE_CLIENT_ID_` and `GOOGLE_CLIENT_SECRET_` from ENV +2. Map scope aliases to Google OAuth scopes +3. Build auth URL with `Signet::OAuth2::Client` (redirect_uri: `urn:ietf:wg:oauth:2.0:oob`) +4. Print URL to console +5. Prompt user to paste authorization code +6. Exchange code for tokens +7. Output JSON to console: + ```json + {"client_id":"...","client_secret":"...","refresh_token":"..."} + ``` +8. Instruct user to store in Vault as `GOOGLE_CREDENTIALS_` + +### `status` flow + +1. Check if `GOOGLE_CREDENTIALS_` exists in ENV +2. Try to fetch access token (validates refresh token) +3. Report: credentials present, token valid/invalid + +### Scope aliases + +| Alias | Google scope | +|-------|-------------| +| `sheets` | `https://www.googleapis.com/auth/spreadsheets.readonly` | +| `gmail` | `https://www.googleapis.com/auth/gmail.send` | + +--- + +## Env Vars + +**Pre-OAuth (temporary, from Google Cloud Console):** +- `GOOGLE_CLIENT_ID_PXOPULSE` +- `GOOGLE_CLIENT_SECRET_PXOPULSE` + +**Post-OAuth (permanent, from `bin/google-oauth authorize`):** +- `GOOGLE_CREDENTIALS_PXOPULSE` — JSON with client_id, client_secret, refresh_token + +All loaded from Vault via `R3x::Env.load_from_vault` at boot. + +--- + +## Vault structure + +``` +secret/data/env/r3x + ├── GOOGLE_CLIENT_ID_PXOPULSE + ├── GOOGLE_CLIENT_SECRET_PXOPULSE + ├── GOOGLE_CREDENTIALS_PXOPULSE + └── GOOGLE_CREDENTIALS_OTHERPROJECT +``` + +--- + +## Per-project pattern + +Follows same naming as LLM API keys (`GEMINI_API_KEY_MICHAL`). + +Workflow specifies which credentials to use: +```ruby +ctx.client.google_sheets( + spreadsheet_id: "...", + credentials_env: "GOOGLE_CREDENTIALS_PXOPULSE" +) +``` + +Validation via `secure_fetch`: +```ruby +R3x::Env.secure_fetch(credentials_env, prefix: "GOOGLE_CREDENTIALS_") +``` + +--- + +## Dependencies + +- `googleauth` gem (already in Gemfile) +- `signet` (comes with googleauth) +- `multi_json` (already in Gemfile) + +--- + +## Related files + +- `lib/r3x/env.rb` — `secure_fetch`, `load_from_vault` +- `config/initializers/r3x_vault_env.rb` — boot-time Vault loading +- `docs/todo/plan-google-sheets-client.md` +- `docs/todo/plan-gmail-output.md` diff --git a/docs/todo/plan-google-sheets-client.md b/docs/todo/plan-google-sheets-client.md new file mode 100644 index 00000000..90c23fa0 --- /dev/null +++ b/docs/todo/plan-google-sheets-client.md @@ -0,0 +1,169 @@ +# Plan: Google Sheets Client + +## Overview + +R3x client for reading Google Sheets data, supporting per-project credentials via Vault. + +## Files + +1. `Gemfile` — add `google-apis-sheets_v4` +2. `app/lib/r3x/client/google_sheets.rb` — client implementation +3. `lib/r3x/workflow/context.rb` — add to ClientProxy + +--- + +## 1. Gemfile + +Add after existing Google gems (line ~28): + +```ruby +gem "google-apis-sheets_v4" +``` + +--- + +## 2. `app/lib/r3x/client/google_sheets.rb` + +### Class: `R3x::Client::GoogleSheets::Client` + +```ruby +module R3x + module Client + module GoogleSheets + class Client + def initialize(spreadsheet_id:, credentials:) + @spreadsheet_id = spreadsheet_id + @credentials = credentials + @service = build_service + end + + def read_rows(range:) + response = service.get_spreadsheet_values(spreadsheet_id, range) + rows = response.values || [] + return [] if rows.empty? + + headers = rows.first + rows.drop(1).map { |row| headers.zip(row).to_h } + end + + private + + attr_reader :spreadsheet_id, :credentials, :service + + def build_service + service = Google::Apis::SheetsV4::SheetsService.new + service.authorization = GoogleAuth.from_json( + credentials, + scope: Google::Apis::SheetsV4::AUTH_SPREADSHEETS_READONLY + ) + service + end + end + end + end +end +``` + +### Public API + +| Method | Returns | Description | +|--------|---------|-------------| +| `read_rows(range:)` | `Array` | First row = headers, rest = data | + +### Behavior + +- First row of range = column headers (used as hash keys) +- Remaining rows = data mapped to hashes +- Empty cells = `""` (not nil) +- Empty sheet = `[]` +- Range format: `"SheetName"` or `"SheetName!A1:D100"` + +### Error handling + +- `Google::Apis::ClientError` (403) — sheet not shared with OAuth account +- `Google::Apis::ClientError` (404) — spreadsheet or sheet not found +- `Signet::AuthorizationError` — refresh token expired + +--- + +## 3. ClientProxy addition + +File: `lib/r3x/workflow/context.rb` + +Add inside `ClientProxy` class, after `llm` method: + +```ruby +def google_sheets(spreadsheet_id:, credentials_env:) + credentials = fetch_google_credentials(credentials_env) + R3x::Client::GoogleSheets::Client.new( + spreadsheet_id: spreadsheet_id, + credentials: credentials + ) +end +``` + +Add as private method on `ClientProxy`: + +```ruby +private + +def fetch_google_credentials(credentials_env) + json = R3x::Env.secure_fetch(credentials_env, prefix: "GOOGLE_CREDENTIALS_") + MultiJson.load(json) +end +``` + +--- + +## Usage in workflow + +```ruby +ctx.client.google_sheets( + spreadsheet_id: "13T1oLQXmhbBYMe0shLs-5aJJsW5Esgx9xaNjSMclubU", + credentials_env: "GOOGLE_CREDENTIALS_PXOPULSE" +).read_rows(range: "ThisWeekApproved") + +# Returns: +# [ +# {"name" => "Concert", "start_date" => "15/1/2026", "location" => "Beach", ...}, +# {"name" => "Festival", "start_date" => "17/1/2026", "location" => "Town", ...} +# ] +``` + +--- + +## PxoWeekly sheet columns + +Based on n8n workflow, `ThisWeekApproved` has: +- `name` — event name +- `start_date` — format `d/M/yyyy` +- `end_date` — optional +- `start_time` — time string +- `location` — venue +- `category` — event category + +--- + +## Sharing requirement + +The Google Sheet must be shared with the OAuth account's email address. +If not shared, API returns 403 `insufficientPermissions`. + +--- + +## Dependencies + +```ruby +gem "google-apis-sheets_v4" # NEW — add to Gemfile +gem "googleauth" # Already present +gem "multi_json" # Already present +``` + +--- + +## Related files + +- `app/lib/r3x/client/google_auth.rb` — shared OAuth2 module (plan: plan-google-oauth2.md) +- `lib/r3x/workflow/context.rb` — ClientProxy integration +- `docs/todo/plan-google-oauth2.md` — OAuth2 setup +- `docs/todo/plan-gmail-output.md` — Gmail output From ef732b487b38063fd9ce6c2ad6eba2a76b650893 Mon Sep 17 00:00:00 2001 From: zewelor Date: Sat, 21 Mar 2026 17:09:20 +0000 Subject: [PATCH 2/7] Make workflows ActiveJob-compatible - Have R3x::Workflow::Base inherit ApplicationJob and include Continuable - Execute workflows with perform(trigger_key, trigger_payload:) calling run(ctx) - Change enqueuing to call workflow_class.perform_later / perform_now - Generate recurring tasks using workflow class name and queue_name - Add guard to prevent overriding #perform; require #run(ctx) in subclasses - Update ChangeDetectionJob, RunWorkflowJob, ManualRunner and tests to match --- app/jobs/r3x/change_detection_job.rb | 8 +--- app/jobs/r3x/run_workflow_job.rb | 26 +---------- lib/r3x/recurring_tasks_config.rb | 28 ++++++++---- lib/r3x/workflow/base.rb | 48 ++++++++++++++++++++- lib/r3x/workflow/manual_runner.rb | 13 +----- test/jobs/r3x/change_detection_job_test.rb | 25 ++++++----- test/lib/r3x/recurring_tasks_config_test.rb | 4 +- test/lib/r3x/workflow_test.rb | 22 +++++++++- 8 files changed, 103 insertions(+), 71 deletions(-) diff --git a/app/jobs/r3x/change_detection_job.rb b/app/jobs/r3x/change_detection_job.rb index f236fe06..6da704c3 100644 --- a/app/jobs/r3x/change_detection_job.rb +++ b/app/jobs/r3x/change_detection_job.rb @@ -19,16 +19,12 @@ def perform(workflow_key, options = nil) TriggerState.transaction do if result[:changed] - R3x::RunWorkflowJob.perform_later( - workflow_key, - trigger_key: trigger_key, - trigger_payload: result[:payload] - ) + workflow_class.perform_later(trigger_key, trigger_payload: result[:payload]) end trigger_state.record_check!(result) end - rescue StandardError => e + rescue => e trigger_state.record_error!(e) if defined?(trigger_state) && trigger_state&.persisted? raise end diff --git a/app/jobs/r3x/run_workflow_job.rb b/app/jobs/r3x/run_workflow_job.rb index 747cfe71..083c1358 100644 --- a/app/jobs/r3x/run_workflow_job.rb +++ b/app/jobs/r3x/run_workflow_job.rb @@ -9,21 +9,7 @@ def perform(workflow_key, options = nil) R3x::Workflow::PackLoader.load! workflow_class = R3x::Workflow::Registry.fetch(workflow_key) - trigger = find_trigger(workflow_class: workflow_class, trigger_key: trigger_key) - - execution = R3x::TriggerManager::Execution.new( - trigger: trigger, - workflow_key: workflow_key, - payload: trigger_payload - ) - - ctx = R3x::Workflow::Context.new( - trigger: execution, - workflow_key: workflow_key, - workflow_class: workflow_class - ) - - workflow_class.new.run(ctx) + workflow_class.new.perform(trigger_key, trigger_payload: trigger_payload) end private @@ -52,15 +38,5 @@ def normalize_options_hash(options) raise ArgumentError, "Expected options hash, got #{options.class.name}" end end - - def find_trigger(workflow_class:, trigger_key:) - trigger = workflow_class.triggers_by_key[trigger_key] - - if trigger.nil? - raise ArgumentError, "Unknown trigger key '#{trigger_key}' for workflow '#{workflow_class.workflow_key}'" - end - - trigger - end end end diff --git a/lib/r3x/recurring_tasks_config.rb b/lib/r3x/recurring_tasks_config.rb index 78bac76d..d4508c85 100644 --- a/lib/r3x/recurring_tasks_config.rb +++ b/lib/r3x/recurring_tasks_config.rb @@ -16,7 +16,7 @@ def schedule_all! triggers.each do |trigger| key = namespaced_key(workflow_key, trigger) current_keys << key - task_options << [ key, task_options_for(workflow_key: workflow_key, trigger: trigger) ] + task_options << [ key, task_options_for(workflow_class: workflow_class, trigger: trigger) ] end end @@ -48,7 +48,7 @@ def to_h triggers.each do |trigger| result_key = namespaced_key(workflow_key, trigger) result[result_key] = task_options_for( - workflow_key: workflow_key, trigger: trigger + workflow_class: workflow_class, trigger: trigger ).stringify_keys end end @@ -62,13 +62,23 @@ def namespaced_key(workflow_key, trigger) "workflow:#{workflow_key}:#{trigger.unique_key}" end - def task_options_for(workflow_key:, trigger:) - { - class: trigger.change_detecting? ? "R3x::ChangeDetectionJob" : "R3x::RunWorkflowJob", - args: [ workflow_key, { "trigger_key" => trigger.unique_key } ], - schedule: trigger.cron, - queue: "default" - } + def task_options_for(workflow_class:, trigger:) + queue_name = workflow_class.new.queue_name + if trigger.change_detecting? + { + class: "R3x::ChangeDetectionJob", + args: [ workflow_class.workflow_key, { "trigger_key" => trigger.unique_key } ], + schedule: trigger.cron, + queue: queue_name + } + else + { + class: workflow_class.name, + args: [ trigger.unique_key ], + schedule: trigger.cron, + queue: queue_name + } + end end end end diff --git a/lib/r3x/workflow/base.rb b/lib/r3x/workflow/base.rb index f7604eff..562c6780 100644 --- a/lib/r3x/workflow/base.rb +++ b/lib/r3x/workflow/base.rb @@ -1,11 +1,55 @@ module R3x module Workflow - class Base + class Base < ApplicationJob + include ActiveJob::Continuable include Dsl include R3x::Concerns::Logger + class << self + def method_added(method_name) + if method_name == :perform && self != Base + raise ArgumentError, "Do not override #perform in #{name}. Override #run(ctx) instead." + end + super + end + end + + def perform(trigger_key, trigger_payload: nil) + R3x::Workflow::PackLoader.load! + trigger = resolve_trigger(trigger_key) + ctx = build_context(trigger: trigger, trigger_payload: trigger_payload) + run(ctx) + end + def run(ctx) - raise NotImplementedError, "Workflow must implement #run(ctx)" + raise NotImplementedError, "#{self.class.name} must implement #run(ctx)" + end + + private + + def resolve_trigger(trigger_key) + trigger = self.class.triggers_by_key[trigger_key] + trigger ||= self.class.triggers.find(&:manual?) + + if trigger.nil? + raise ArgumentError, "Unknown trigger key '#{trigger_key}' for workflow '#{self.class.workflow_key}'" + end + + trigger + end + + def build_context(trigger:, trigger_payload:) + execution = R3x::TriggerManager::Execution.new( + trigger: trigger, + workflow_key: self.class.workflow_key, + payload: trigger_payload + ) + + Context.new( + trigger: execution, + workflow_key: self.class.workflow_key, + workflow_class: self.class + ) end end end diff --git a/lib/r3x/workflow/manual_runner.rb b/lib/r3x/workflow/manual_runner.rb index 55a8fcc0..7c49904f 100644 --- a/lib/r3x/workflow/manual_runner.rb +++ b/lib/r3x/workflow/manual_runner.rb @@ -13,18 +13,7 @@ def run(workflow_key) workflow_class = Registry.fetch(workflow_key) trigger = workflow_class.triggers.find(&:manual?) || Triggers::Manual.new - execution = TriggerManager::Execution.new( - trigger: trigger, - workflow_key: workflow_key - ) - - ctx = Context.new( - trigger: execution, - workflow_key: workflow_key, - workflow_class: workflow_class - ) - - workflow_class.new.run(ctx) + workflow_class.new.perform_now(trigger.unique_key) end end end diff --git a/test/jobs/r3x/change_detection_job_test.rb b/test/jobs/r3x/change_detection_job_test.rb index 92415965..2d2ee45c 100644 --- a/test/jobs/r3x/change_detection_job_test.rb +++ b/test/jobs/r3x/change_detection_job_test.rb @@ -53,20 +53,18 @@ class ChangeDetectionJobTest < ActiveSupport::TestCase end ) - register_change_detecting_workflow(fake_trigger) + workflow_class = register_change_detecting_workflow(fake_trigger) - assert_enqueued_jobs 1, only: R3x::RunWorkflowJob do + assert_enqueued_jobs 1, only: workflow_class do ChangeDetectionJob.perform_now("test_change_detecting_feed", { "trigger_key" => fake_trigger.unique_key }) end enqueued_job = enqueued_jobs.last - assert_equal R3x::RunWorkflowJob, enqueued_job[:job] - assert_equal "test_change_detecting_feed", enqueued_job[:args][0] - assert_equal fake_trigger.unique_key, enqueued_job[:args][1]["trigger_key"] - assert_equal( - { "entries" => [ { "title" => "Hello", "_aj_symbol_keys" => [ "title" ] } ], "_aj_symbol_keys" => [ "entries" ] }, - enqueued_job[:args][1]["trigger_payload"] - ) + assert_equal workflow_class, enqueued_job[:job] + assert_equal fake_trigger.unique_key, enqueued_job[:args][0] + payload = enqueued_job[:args][1]["trigger_payload"] + assert_equal 1, payload["entries"].length + assert_equal "Hello", payload["entries"].first["title"] state = R3x::TriggerState.find_by!(workflow_key: "test_change_detecting_feed", trigger_key: fake_trigger.unique_key) assert_equal({ "cursor" => "v2" }, state.state) @@ -81,17 +79,17 @@ class ChangeDetectionJobTest < ActiveSupport::TestCase end ) - register_change_detecting_workflow(fake_trigger) + workflow_class = register_change_detecting_workflow(fake_trigger) - original_perform_later = R3x::RunWorkflowJob.method(:perform_later) - R3x::RunWorkflowJob.singleton_class.send(:define_method, :perform_later) do |*| + original_perform_later = workflow_class.method(:perform_later) + workflow_class.singleton_class.send(:define_method, :perform_later) do |*| raise ActiveJob::EnqueueError, "enqueue failed" end error = assert_raises(ActiveJob::EnqueueError) do ChangeDetectionJob.perform_now("test_change_detecting_feed", { "trigger_key" => fake_trigger.unique_key }) ensure - R3x::RunWorkflowJob.singleton_class.send(:define_method, :perform_later, original_perform_later) + workflow_class.singleton_class.send(:define_method, :perform_later, original_perform_later) end assert_equal "enqueue failed", error.message @@ -142,6 +140,7 @@ def run(ctx) end Workflow::Registry.register(workflow_class) + workflow_class end end end diff --git a/test/lib/r3x/recurring_tasks_config_test.rb b/test/lib/r3x/recurring_tasks_config_test.rb index 7844b479..ce44292c 100644 --- a/test/lib/r3x/recurring_tasks_config_test.rb +++ b/test/lib/r3x/recurring_tasks_config_test.rb @@ -21,7 +21,7 @@ class RecurringTasksConfigTest < ActiveSupport::TestCase task = tasks.find { |k, _| k.start_with?(expected_key) }&.last assert task, "Expected task with key starting with #{expected_key}" - assert_equal "R3x::RunWorkflowJob", task["class"] + assert_equal "Workflows::TestWorkflow", task["class"] assert_equal "0 * * * *", task["schedule"] assert_equal "default", task["queue"] end @@ -76,7 +76,7 @@ def self.name task = dynamic_tasks.find { |t| t.key.include?(":schedule:") } assert task, "Expected a schedule trigger task" - assert_equal "R3x::RunWorkflowJob", task.class_name + assert_equal "Workflows::TestWorkflow", task.class_name assert_equal "0 * * * *", task.schedule end diff --git a/test/lib/r3x/workflow_test.rb b/test/lib/r3x/workflow_test.rb index eece99cb..da07645d 100644 --- a/test/lib/r3x/workflow_test.rb +++ b/test/lib/r3x/workflow_test.rb @@ -1,4 +1,5 @@ require "test_helper" +require_relative "../../support/fake_change_detecting_trigger" module R3x class WorkflowTest < ActiveSupport::TestCase @@ -232,12 +233,29 @@ def self.name "Workflows::WithTriggers" end - trigger :schedule, cron: "0 12 * * *" + trigger :manual end triggers = klass.triggers assert_equal 1, triggers.size - assert_equal :schedule, triggers.first.type + assert_equal :manual, triggers.first.type + end + + test "prevents overriding perform method in subclasses" do + error = assert_raises(ArgumentError) do + Class.new(R3x::Workflow::Base) do + def self.name + "Workflows::BadWorkflow" + end + + def perform + # This should raise an error + end + end + end + + assert_match(/Do not override #perform/, error.message) + assert_match(/Override #run\(ctx\) instead/, error.message) end test "schedulable_triggers excludes auto-generated Manual triggers" do From 5c615af52edef659a5891acff6d26c7e88ddc046 Mon Sep 17 00:00:00 2001 From: zewelor Date: Fri, 27 Mar 2026 13:47:32 +0100 Subject: [PATCH 3/7] Remove Gmail output plan doc - Remove draft plan file `docs/todo/plan-gmail-output.md` - Remove stale design notes for Gmail output implementation - Reduce noise in `docs/todo`; plan is implemented or relocated --- docs/todo/plan-gmail-output.md | 228 --------------------------------- 1 file changed, 228 deletions(-) delete mode 100644 docs/todo/plan-gmail-output.md diff --git a/docs/todo/plan-gmail-output.md b/docs/todo/plan-gmail-output.md deleted file mode 100644 index 180944c0..00000000 --- a/docs/todo/plan-gmail-output.md +++ /dev/null @@ -1,228 +0,0 @@ -# Plan: Gmail Output - -## Overview - -R3x output for sending emails via Gmail API with test mode for safe development. -Follows the `R3x::Outputs::Discord` pattern. - -## Files - -1. `Gemfile` — add `mail` gem (if not present) -2. `app/lib/r3x/outputs/gmail.rb` — output implementation -3. `lib/r3x/workflow/context.rb` — add to ClientProxy - ---- - -## 1. Gemfile - -Check if `mail` gem is present, if not add: - -```ruby -gem "mail" # For RFC 2822 email building -``` - ---- - -## 2. `app/lib/r3x/outputs/gmail.rb` - -### Class: `R3x::Outputs::Gmail` - -```ruby -module R3x - module Outputs - class Gmail - include R3x::Concerns::Logger - - def initialize(credentials:, mode: nil) - @credentials = credentials - @mode = mode || ENV.fetch("R3X_GMAIL_MODE", "test") - end - - def deliver(to:, subject:, body:) - case mode - when "real" - deliver_real(to: to, subject: subject, body: body) - when "test" - deliver_test(to: to, subject: subject, body: body) - else - raise ArgumentError, "Unsupported Gmail mode: #{mode}. Supported: real, test" - end - end - - private - - attr_reader :credentials, :mode - - def deliver_real(to:, subject:, body:) - service = Google::Apis::GmailV1::GmailService.new - service.authorization = R3x::Client::GoogleAuth.from_json( - credentials, - scope: Google::Apis::GmailV1::AUTH_GMAIL_SEND - ) - - mail = Mail.new do - to to - subject subject - body body - end - - message = Google::Apis::GmailV1::Message.new( - raw: Base64.urlsafe_encode64(mail.to_s) - ) - - result = service.send_user_message("me", message) - {"mode" => "real", "message_id" => result.id} - end - - def deliver_test(to:, subject:, body:) - logger.info("Gmail [TEST] to=#{to} subject=#{subject}\n#{body}") - {"mode" => "test"} - end - end - end -end -``` - -### Public API - -| Method | Returns | Description | -|--------|---------|-------------| -| `deliver(to:, subject:, body:)` | `Hash` | Send email (real mode) or log (test mode) | - -### Mode behavior - -| Mode | Env var value | Behavior | -|------|--------------|----------| -| `test` | default | Log to Rails logger, no API call | -| `real` | `"real"` | Send via Gmail API | - -### Error handling - -- `Google::Apis::ClientError` (400) — invalid recipient -- `Google::Apis::ClientError` (403) — quota exceeded or permission denied -- `Signet::AuthorizationError` — refresh token expired - ---- - -## 3. ClientProxy addition - -File: `lib/r3x/workflow/context.rb` - -Add inside `ClientProxy` class: - -```ruby -def gmail(credentials_env:, mode: nil) - credentials = fetch_google_credentials(credentials_env) - R3x::Outputs::Gmail.new(credentials: credentials, mode: mode) -end -``` - -The `fetch_google_credentials` private method is shared with `google_sheets` (see plan-google-sheets-client.md). - ---- - -## Env var - -```bash -R3X_GMAIL_MODE=test # or "real" for production -``` - -Can also be overridden per-call: -```ruby -ctx.client.gmail(credentials_env: "...", mode: "real").deliver(...) -``` - ---- - -## Usage in workflow - -```ruby -ctx.client.gmail( - credentials_env: "GOOGLE_CREDENTIALS_PXOPULSE" -).deliver( - to: "pxopulse@gmail.com", - subject: "Weekly pulse", - body: formatted_content -) - -# Test mode returns: -# {"mode" => "test"} - -# Real mode returns: -# {"mode" => "real", "message_id" => "18a3f..."} -``` - ---- - -## PxoWeekly email format - -Based on n8n workflow output: - -``` -🇬🇧🎉 This week in Porto Santo - -🌅 #Sunsessions with M da Silva -📍 Foot On Water Restaurant & Beach Bar -📅 17 de agosto, 16h30–20h30 - -🎤 Concert at Praça -📍 Town Square -📅 18 de agosto, 21h00 - -💬 More details on pxopulse.com - - -============== - -🇵🇹🎉 Esta semana em Porto Santo - -🌅 #Sunsessions com M da Silva -📍 Foot On Water Restaurant & Beach Bar -📅 17 de agosto, 16h30–20h30 - -🎤 Concerto na Praça -📍 Praça do Povo -📅 18 de agosto, 21h00 - -💬 Mais detalhes em pxopulse.com -``` - ---- - -## Comparison with Discord Output - -| Aspect | `Outputs::Discord` | `Outputs::Gmail` | -|--------|-------------------|------------------| -| Mode env var | `R3X_DISCORD_MODE` | `R3X_GMAIL_MODE` | -| Test output | Logs to Rails logger | Logs to Rails logger | -| Auth | Webhook URL | OAuth2 credentials | -| Content | Plain text | RFC 2822 email | -| Method | `deliver(content:)` | `deliver(to:, subject:, body:)` | - ---- - -## Gmail API limits - -- Personal account: ~100 emails/day -- Google Workspace: ~1500 emails/day -- OAuth app must be in "Testing" or "Published" status - ---- - -## Dependencies - -```ruby -gem "google-apis-gmail_v1" # Already in Gemfile -gem "mail" # Check/add for email building -gem "googleauth" # Already in Gemfile -``` - ---- - -## Related files - -- `app/lib/r3x/client/google_auth.rb` — shared OAuth2 module (plan: plan-google-oauth2.md) -- `app/lib/r3x/outputs/discord.rb` — similar output pattern -- `lib/r3x/workflow/context.rb` — ClientProxy integration -- `docs/todo/plan-google-oauth2.md` — OAuth2 setup -- `docs/todo/plan-google-sheets-client.md` — Sheets client From 0f4d0c73aaf6cadc47c7464564f693026ae43b5c Mon Sep 17 00:00:00 2001 From: zewelor Date: Fri, 27 Mar 2026 13:48:21 +0100 Subject: [PATCH 4/7] Remove Google OAuth2 and Sheets plan docs - Remove stale TODO plan docs for Google integrations - Delete `docs/todo/plan-google-oauth2.md` - Delete `docs/todo/plan-google-sheets-client.md` - Tidy docs by removing outdated planning notes --- docs/todo/plan-google-oauth2.md | 154 ---------------------- docs/todo/plan-google-sheets-client.md | 169 ------------------------- 2 files changed, 323 deletions(-) delete mode 100644 docs/todo/plan-google-oauth2.md delete mode 100644 docs/todo/plan-google-sheets-client.md diff --git a/docs/todo/plan-google-oauth2.md b/docs/todo/plan-google-oauth2.md deleted file mode 100644 index 93d57eb6..00000000 --- a/docs/todo/plan-google-oauth2.md +++ /dev/null @@ -1,154 +0,0 @@ -# Plan: Google OAuth2 Helper - -## Overview - -Interactive CLI helper for obtaining Google refresh tokens and shared OAuth2 credential module -for per-project Google API credentials stored in Vault. - -## Files - -1. `app/lib/r3x/client/google_auth.rb` — shared credential builder -2. `bin/google-oauth` — interactive CLI helper - ---- - -## 1. `app/lib/r3x/client/google_auth.rb` - -Shared module for building OAuth2 credentials from JSON. - -```ruby -module R3x::Client::GoogleAuth - SCOPE_ALIASES = { - sheets: Google::Apis::SheetsV4::AUTH_SPREADSHEETS_READONLY, - gmail: Google::Apis::GmailV1::AUTH_GMAIL_SEND - }.freeze - - def self.from_json(parsed_json, scope:) - Signet::OAuth2::Client.new( - client_id: parsed_json.fetch("client_id"), - client_secret: parsed_json.fetch("client_secret"), - refresh_token: parsed_json.fetch("refresh_token"), - token_credential_uri: "https://oauth2.googleapis.com/token", - scope: Array(scope) - ).tap(&:fetch_access_token!) - end -end -``` - -**Error handling:** -- `KeyError` if required fields missing from JSON -- Propagate `Signet::AuthorizationError` on invalid refresh token - ---- - -## 2. `bin/google-oauth` - -```bash -#!/usr/bin/env ruby -require_relative "../config/environment" -require "optparse" -``` - -### Commands - -| Command | Description | -|---------|-------------| -| `authorize --project PROJECT --scopes sheets,gmail` | Start OAuth2 flow | -| `status --project PROJECT` | Check credential status | - -### Pre-setup (manual) - -1. Google Cloud Console → create project -2. Enable APIs: Sheets API + Gmail API -3. OAuth consent screen → External → Add test users -4. Credentials → Create OAuth client ID → Desktop app -5. Extract `client_id` and `client_secret` -6. Store in Vault: `GOOGLE_CLIENT_ID_`, `GOOGLE_CLIENT_SECRET_` - -### `authorize` flow - -1. Read `GOOGLE_CLIENT_ID_` and `GOOGLE_CLIENT_SECRET_` from ENV -2. Map scope aliases to Google OAuth scopes -3. Build auth URL with `Signet::OAuth2::Client` (redirect_uri: `urn:ietf:wg:oauth:2.0:oob`) -4. Print URL to console -5. Prompt user to paste authorization code -6. Exchange code for tokens -7. Output JSON to console: - ```json - {"client_id":"...","client_secret":"...","refresh_token":"..."} - ``` -8. Instruct user to store in Vault as `GOOGLE_CREDENTIALS_` - -### `status` flow - -1. Check if `GOOGLE_CREDENTIALS_` exists in ENV -2. Try to fetch access token (validates refresh token) -3. Report: credentials present, token valid/invalid - -### Scope aliases - -| Alias | Google scope | -|-------|-------------| -| `sheets` | `https://www.googleapis.com/auth/spreadsheets.readonly` | -| `gmail` | `https://www.googleapis.com/auth/gmail.send` | - ---- - -## Env Vars - -**Pre-OAuth (temporary, from Google Cloud Console):** -- `GOOGLE_CLIENT_ID_PXOPULSE` -- `GOOGLE_CLIENT_SECRET_PXOPULSE` - -**Post-OAuth (permanent, from `bin/google-oauth authorize`):** -- `GOOGLE_CREDENTIALS_PXOPULSE` — JSON with client_id, client_secret, refresh_token - -All loaded from Vault via `R3x::Env.load_from_vault` at boot. - ---- - -## Vault structure - -``` -secret/data/env/r3x - ├── GOOGLE_CLIENT_ID_PXOPULSE - ├── GOOGLE_CLIENT_SECRET_PXOPULSE - ├── GOOGLE_CREDENTIALS_PXOPULSE - └── GOOGLE_CREDENTIALS_OTHERPROJECT -``` - ---- - -## Per-project pattern - -Follows same naming as LLM API keys (`GEMINI_API_KEY_MICHAL`). - -Workflow specifies which credentials to use: -```ruby -ctx.client.google_sheets( - spreadsheet_id: "...", - credentials_env: "GOOGLE_CREDENTIALS_PXOPULSE" -) -``` - -Validation via `secure_fetch`: -```ruby -R3x::Env.secure_fetch(credentials_env, prefix: "GOOGLE_CREDENTIALS_") -``` - ---- - -## Dependencies - -- `googleauth` gem (already in Gemfile) -- `signet` (comes with googleauth) -- `multi_json` (already in Gemfile) - ---- - -## Related files - -- `lib/r3x/env.rb` — `secure_fetch`, `load_from_vault` -- `config/initializers/r3x_vault_env.rb` — boot-time Vault loading -- `docs/todo/plan-google-sheets-client.md` -- `docs/todo/plan-gmail-output.md` diff --git a/docs/todo/plan-google-sheets-client.md b/docs/todo/plan-google-sheets-client.md deleted file mode 100644 index 90c23fa0..00000000 --- a/docs/todo/plan-google-sheets-client.md +++ /dev/null @@ -1,169 +0,0 @@ -# Plan: Google Sheets Client - -## Overview - -R3x client for reading Google Sheets data, supporting per-project credentials via Vault. - -## Files - -1. `Gemfile` — add `google-apis-sheets_v4` -2. `app/lib/r3x/client/google_sheets.rb` — client implementation -3. `lib/r3x/workflow/context.rb` — add to ClientProxy - ---- - -## 1. Gemfile - -Add after existing Google gems (line ~28): - -```ruby -gem "google-apis-sheets_v4" -``` - ---- - -## 2. `app/lib/r3x/client/google_sheets.rb` - -### Class: `R3x::Client::GoogleSheets::Client` - -```ruby -module R3x - module Client - module GoogleSheets - class Client - def initialize(spreadsheet_id:, credentials:) - @spreadsheet_id = spreadsheet_id - @credentials = credentials - @service = build_service - end - - def read_rows(range:) - response = service.get_spreadsheet_values(spreadsheet_id, range) - rows = response.values || [] - return [] if rows.empty? - - headers = rows.first - rows.drop(1).map { |row| headers.zip(row).to_h } - end - - private - - attr_reader :spreadsheet_id, :credentials, :service - - def build_service - service = Google::Apis::SheetsV4::SheetsService.new - service.authorization = GoogleAuth.from_json( - credentials, - scope: Google::Apis::SheetsV4::AUTH_SPREADSHEETS_READONLY - ) - service - end - end - end - end -end -``` - -### Public API - -| Method | Returns | Description | -|--------|---------|-------------| -| `read_rows(range:)` | `Array` | First row = headers, rest = data | - -### Behavior - -- First row of range = column headers (used as hash keys) -- Remaining rows = data mapped to hashes -- Empty cells = `""` (not nil) -- Empty sheet = `[]` -- Range format: `"SheetName"` or `"SheetName!A1:D100"` - -### Error handling - -- `Google::Apis::ClientError` (403) — sheet not shared with OAuth account -- `Google::Apis::ClientError` (404) — spreadsheet or sheet not found -- `Signet::AuthorizationError` — refresh token expired - ---- - -## 3. ClientProxy addition - -File: `lib/r3x/workflow/context.rb` - -Add inside `ClientProxy` class, after `llm` method: - -```ruby -def google_sheets(spreadsheet_id:, credentials_env:) - credentials = fetch_google_credentials(credentials_env) - R3x::Client::GoogleSheets::Client.new( - spreadsheet_id: spreadsheet_id, - credentials: credentials - ) -end -``` - -Add as private method on `ClientProxy`: - -```ruby -private - -def fetch_google_credentials(credentials_env) - json = R3x::Env.secure_fetch(credentials_env, prefix: "GOOGLE_CREDENTIALS_") - MultiJson.load(json) -end -``` - ---- - -## Usage in workflow - -```ruby -ctx.client.google_sheets( - spreadsheet_id: "13T1oLQXmhbBYMe0shLs-5aJJsW5Esgx9xaNjSMclubU", - credentials_env: "GOOGLE_CREDENTIALS_PXOPULSE" -).read_rows(range: "ThisWeekApproved") - -# Returns: -# [ -# {"name" => "Concert", "start_date" => "15/1/2026", "location" => "Beach", ...}, -# {"name" => "Festival", "start_date" => "17/1/2026", "location" => "Town", ...} -# ] -``` - ---- - -## PxoWeekly sheet columns - -Based on n8n workflow, `ThisWeekApproved` has: -- `name` — event name -- `start_date` — format `d/M/yyyy` -- `end_date` — optional -- `start_time` — time string -- `location` — venue -- `category` — event category - ---- - -## Sharing requirement - -The Google Sheet must be shared with the OAuth account's email address. -If not shared, API returns 403 `insufficientPermissions`. - ---- - -## Dependencies - -```ruby -gem "google-apis-sheets_v4" # NEW — add to Gemfile -gem "googleauth" # Already present -gem "multi_json" # Already present -``` - ---- - -## Related files - -- `app/lib/r3x/client/google_auth.rb` — shared OAuth2 module (plan: plan-google-oauth2.md) -- `lib/r3x/workflow/context.rb` — ClientProxy integration -- `docs/todo/plan-google-oauth2.md` — OAuth2 setup -- `docs/todo/plan-gmail-output.md` — Gmail output From 9128702bc0668b891d3bce4f0b229e6a6746c476 Mon Sep 17 00:00:00 2001 From: zewelor Date: Fri, 27 Mar 2026 19:05:48 +0100 Subject: [PATCH 5/7] Tweaks --- AGENTS.md | 10 +-- app/jobs/r3x/run_workflow_job.rb | 2 +- lib/r3x/workflow/base.rb | 38 ++-------- lib/r3x/workflow/executor.rb | 46 ++++++++++++ lib/r3x/workflow/manual_runner.rb | 3 +- test/jobs/r3x/run_workflow_job_test.rb | 36 ++++++++++ test/lib/r3x/workflow/manual_runner_test.rb | 40 +++++++++++ test/lib/r3x/workflow_test.rb | 79 +++++++++++++++++++++ 8 files changed, 216 insertions(+), 38 deletions(-) create mode 100644 lib/r3x/workflow/executor.rb create mode 100644 test/lib/r3x/workflow/manual_runner_test.rb diff --git a/AGENTS.md b/AGENTS.md index b239c127..b062f071 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -15,13 +15,14 @@ This Rails app uses a small set of preferred libraries for common integration wo ## Codebase Map - `lib/r3x/`: core framework code for the workflow DSL, trigger types, workflow loading, registry, execution context, recurring-task config, and shared DSL helpers. +- `lib/r3x/workflow/executor.rb`: shared workflow execution helper that resolves the trigger and builds `Workflow::Context` for a loaded workflow class. - `lib/r3x/dsl/`: shared DSL infrastructure, especially validation concerns and configuration errors used by workflow-declared objects. - `lib/r3x/trigger_manager.rb` + `lib/r3x/trigger_manager/`: trigger infrastructure — `R3x::TriggerManager::Collection` (manages workflow triggers as a hash keyed by `unique_key`) and `R3x::TriggerManager::Execution` (wraps a trigger for runtime use). - `app/lib/r3x/`: runtime support code such as outputs, client wrappers, and shared concerns. - `app/lib/r3x/client/google/credentials.rb`: shared Google credentials loader used by Gmail and Google Sheets integrations. - `app/lib/r3x/client/google/gmail.rb`: Gmail API client used by `R3x::Outputs::Gmail`. - `R3x::Client::Google` is a project namespace; when referencing the third-party Google gem namespace, use `::Google` to avoid constant collisions. -- `app/jobs/r3x/`: job entrypoints, especially `R3x::RunWorkflowJob`, which resolves and executes workflows, and `R3x::ChangeDetectionJob`, which evaluates change-detecting triggers before enqueueing workflow runs. +- `app/jobs/r3x/`: job entrypoints, especially `R3x::RunWorkflowJob`, which resolves a workflow key and dispatches to the workflow job class, and `R3x::ChangeDetectionJob`, which evaluates change-detecting triggers before enqueueing workflow runs. - `app/models/r3x/`: runtime support models such as `R3x::TriggerState` for per-trigger change-detection state. - `workflows/`: user workflow packs. These are not the framework itself; they are loaded by the framework. - `config/initializers/r3x_workflow_loader.rb`: boot-time workflow loading hook. @@ -30,13 +31,14 @@ This Rails app uses a small set of preferred libraries for common integration wo ## Runtime Flow - Workflows subclass `R3x::Workflow::Base`, declare triggers via the DSL, and implement `#run(ctx)`. +- `R3x::Workflow::Base` is also an `ApplicationJob`; its `#perform` delegates trigger/context setup to `R3x::Workflow::Executor` and then calls `#run(ctx)` on the current job instance. - Workflow-declared DSL objects must validate themselves before being registered; invalid DSL configuration should raise `R3x::ConfigurationError` with collected validation errors. - `R3x::Workflow::PackLoader` discovers workflow entrypoints named `workflow.rb` from directories listed in `R3X_WORKFLOW_PATHS`, loads them, and registers their classes in `R3x::Workflow::Registry`. - `R3x::RecurringTasksConfig` turns schedulable workflow triggers into Solid Queue dynamic recurring tasks via `SolidQueue::RecurringTask`. All triggers have a `unique_key` (based on type + options hash) used for identification and duplicate detection. `schedule_all!` persists dynamic tasks and sweeps stale ones. - Change-detecting triggers are file-defined trigger objects that provide `cron`, `unique_key`, and `detect_changes(workflow_key:, state:)`. Their durable runtime state lives in `R3x::TriggerState`. -- `R3x::ChangeDetectionJob` loads the trigger, fetches/updates `R3x::TriggerState`, and only enqueues `R3x::RunWorkflowJob` when the trigger reports a change. +- `R3x::ChangeDetectionJob` loads the trigger, fetches/updates `R3x::TriggerState`, and only enqueues the workflow job class itself when the trigger reports a change. - Because the app currently uses `Solid Queue` as a database-backed backend on the same Active Record database connection, code may intentionally rely on a database transaction covering both `TriggerState` updates and `perform_later`. Do not assume those guarantees survive a future backend or database split. -- `R3x::RunWorkflowJob` fetches the workflow from the registry, resolves the trigger by `trigger_key`, builds a `Workflow::Context`, and calls `workflow_class.new.run(ctx)`. +- `R3x::RunWorkflowJob` fetches the workflow from the registry and calls `workflow_class.perform_now(trigger_key, trigger_payload: ...)` for compatibility with callers that still dispatch by workflow key. - Trigger discovery is filesystem-backed through `lib/r3x/triggers/*.rb`, so trigger file names, constants, and supported types must stay aligned. ## Working with Workflows @@ -71,7 +73,7 @@ bin/workflow [options] [command] [arguments] **Global options:** `-h, --help` — print usage. -`ManualRunner` (`lib/r3x/workflow/manual_runner.rb`) fetches the workflow class from the registry, picks its first manual trigger (or creates a default `Triggers::Manual`), builds a `Workflow::Context`, and calls `workflow_class.new.run(ctx)`. This is the same execution path used by `R3x::RunWorkflowJob` but without Solid Queue — it runs synchronously in the current process. +`ManualRunner` (`lib/r3x/workflow/manual_runner.rb`) fetches the workflow class from the registry and calls `workflow_class.perform_now`. Manual execution is resolved inside the workflow execution path, so workflows can still be run manually even when they only declare non-manual triggers. ### Rake equivalents diff --git a/app/jobs/r3x/run_workflow_job.rb b/app/jobs/r3x/run_workflow_job.rb index 083c1358..d7ff4134 100644 --- a/app/jobs/r3x/run_workflow_job.rb +++ b/app/jobs/r3x/run_workflow_job.rb @@ -9,7 +9,7 @@ def perform(workflow_key, options = nil) R3x::Workflow::PackLoader.load! workflow_class = R3x::Workflow::Registry.fetch(workflow_key) - workflow_class.new.perform(trigger_key, trigger_payload: trigger_payload) + workflow_class.perform_now(trigger_key, trigger_payload: trigger_payload) end private diff --git a/lib/r3x/workflow/base.rb b/lib/r3x/workflow/base.rb index 562c6780..d8fb8fd2 100644 --- a/lib/r3x/workflow/base.rb +++ b/lib/r3x/workflow/base.rb @@ -14,43 +14,19 @@ def method_added(method_name) end end - def perform(trigger_key, trigger_payload: nil) - R3x::Workflow::PackLoader.load! - trigger = resolve_trigger(trigger_key) - ctx = build_context(trigger: trigger, trigger_payload: trigger_payload) + def perform(trigger_key = nil, trigger_payload: nil) + ctx = Executor.build_context( + workflow_class: self.class, + trigger_key: trigger_key, + trigger_payload: trigger_payload + ) + run(ctx) end def run(ctx) raise NotImplementedError, "#{self.class.name} must implement #run(ctx)" end - - private - - def resolve_trigger(trigger_key) - trigger = self.class.triggers_by_key[trigger_key] - trigger ||= self.class.triggers.find(&:manual?) - - if trigger.nil? - raise ArgumentError, "Unknown trigger key '#{trigger_key}' for workflow '#{self.class.workflow_key}'" - end - - trigger - end - - def build_context(trigger:, trigger_payload:) - execution = R3x::TriggerManager::Execution.new( - trigger: trigger, - workflow_key: self.class.workflow_key, - payload: trigger_payload - ) - - Context.new( - trigger: execution, - workflow_key: self.class.workflow_key, - workflow_class: self.class - ) - end end end end diff --git a/lib/r3x/workflow/executor.rb b/lib/r3x/workflow/executor.rb new file mode 100644 index 00000000..36489548 --- /dev/null +++ b/lib/r3x/workflow/executor.rb @@ -0,0 +1,46 @@ +module R3x + module Workflow + class Executor + def self.build_context(...) + new(...).build_context + end + + def initialize(workflow_class:, trigger_key:, trigger_payload: nil) + @workflow_class = workflow_class + @trigger_key = trigger_key + @trigger_payload = trigger_payload + end + + def build_context + Context.new( + trigger: TriggerManager::Execution.new( + trigger: resolve_trigger, + workflow_key: workflow_class.workflow_key, + payload: trigger_payload + ), + workflow_key: workflow_class.workflow_key, + workflow_class: workflow_class + ) + end + + private + + attr_reader :workflow_class, :trigger_key, :trigger_payload + + def resolve_trigger + return manual_trigger if trigger_key.nil? + return manual_trigger if manual_trigger.unique_key == trigger_key + + workflow_class.triggers_by_key[trigger_key] || unknown_trigger! + end + + def manual_trigger + workflow_class.triggers.find(&:manual?) || Triggers::Manual.new + end + + def unknown_trigger! + raise ArgumentError, "Unknown trigger key '#{trigger_key}' for workflow '#{workflow_class.workflow_key}'" + end + end + end +end diff --git a/lib/r3x/workflow/manual_runner.rb b/lib/r3x/workflow/manual_runner.rb index 7c49904f..63afcc63 100644 --- a/lib/r3x/workflow/manual_runner.rb +++ b/lib/r3x/workflow/manual_runner.rb @@ -11,9 +11,8 @@ def initialize def run(workflow_key) workflow_class = Registry.fetch(workflow_key) - trigger = workflow_class.triggers.find(&:manual?) || Triggers::Manual.new - workflow_class.new.perform_now(trigger.unique_key) + workflow_class.perform_now end end end diff --git a/test/jobs/r3x/run_workflow_job_test.rb b/test/jobs/r3x/run_workflow_job_test.rb index 63af5ae5..e95b8539 100644 --- a/test/jobs/r3x/run_workflow_job_test.rb +++ b/test/jobs/r3x/run_workflow_job_test.rb @@ -75,6 +75,42 @@ def run(ctx) Workflow::PackLoader.load!(force: true) end + test "executes workflow through Active Job perform_now" do + job = RunWorkflowJob.new + called = nil + + workflow_class = Class.new(R3x::Workflow::Base) do + def self.name + "TestPerformNow" + end + + trigger :manual + + def run(_ctx) + raise "should not be called directly in this test" + end + end + + Workflow::Registry.register(workflow_class) + manual_trigger = workflow_class.triggers.first + original_perform_now = workflow_class.method(:perform_now) + + workflow_class.singleton_class.send(:define_method, :perform_now) do |*args, **kwargs| + called = { args: args, kwargs: kwargs } + { "mode" => "perform_now" } + end + + result = job.perform("test_perform_now", trigger_key: manual_trigger.unique_key) + + assert_equal({ "mode" => "perform_now" }, result) + assert_equal [ manual_trigger.unique_key ], called[:args] + assert_equal({ trigger_payload: nil }, called[:kwargs]) + ensure + workflow_class.singleton_class.send(:define_method, :perform_now, original_perform_now) if workflow_class && original_perform_now + Workflow::Registry.reset! + Workflow::PackLoader.load!(force: true) + end + test "performs workflow with change-detecting trigger and payload" do job = RunWorkflowJob.new fake_trigger = R3x::TestSupport::FakeChangeDetectingTrigger.new(identity: "feed") diff --git a/test/lib/r3x/workflow/manual_runner_test.rb b/test/lib/r3x/workflow/manual_runner_test.rb new file mode 100644 index 00000000..806612d7 --- /dev/null +++ b/test/lib/r3x/workflow/manual_runner_test.rb @@ -0,0 +1,40 @@ +require "test_helper" + +module R3x + module Workflow + class ManualRunnerTest < ActiveSupport::TestCase + setup do + @original_workflow_paths = ENV["R3X_WORKFLOW_PATHS"] + ENV["R3X_WORKFLOW_PATHS"] = Rails.root.join("test/fixtures/workflows").to_s + + Workflow::PackLoader.load!(force: true) + end + + teardown do + ENV["R3X_WORKFLOW_PATHS"] = @original_workflow_paths + Workflow::Registry.reset! + Workflow::PackLoader.load!(force: true) + end + + test "runs schedule-only workflow through manual trigger" do + workflow_class = Class.new(R3x::Workflow::Base) do + def self.name + "Workflows::ScheduleOnlyManualRun" + end + + trigger :schedule, cron: "0 * * * *" + + def run(ctx) + { "trigger_type" => ctx.trigger.type.to_s } + end + end + + Workflow::Registry.register(workflow_class) + + result = ManualRunner.run("schedule_only_manual_run") + + assert_equal "manual", result["trigger_type"] + end + end + end +end diff --git a/test/lib/r3x/workflow_test.rb b/test/lib/r3x/workflow_test.rb index da07645d..bf367e50 100644 --- a/test/lib/r3x/workflow_test.rb +++ b/test/lib/r3x/workflow_test.rb @@ -241,6 +241,85 @@ def self.name assert_equal :manual, triggers.first.type end + test "perform does not fallback to manual trigger for unknown trigger key" do + workflow_class = Class.new(R3x::Workflow::Base) do + def self.name + "Workflows::StrictTriggerLookup" + end + + trigger :manual + + def run(_ctx) + raise "should not execute" + end + end + + error = assert_raises(ArgumentError) do + workflow_class.perform_now("missing-trigger") + end + + assert_match(/Unknown trigger key 'missing-trigger'/, error.message) + end + + test "perform accepts auto-generated manual trigger when no triggers are declared" do + workflow_class = Class.new(R3x::Workflow::Base) do + def self.name + "Workflows::ImplicitManual" + end + + def run(ctx) + { "trigger_type" => ctx.trigger.type.to_s } + end + end + + result = workflow_class.perform_now(workflow_class.triggers.first.unique_key) + + assert_equal "manual", result["trigger_type"] + end + + test "perform without trigger key uses manual trigger for schedule-only workflow" do + workflow_class = Class.new(R3x::Workflow::Base) do + def self.name + "Workflows::ManualFallback" + end + + trigger :schedule, cron: "0 * * * *" + + def run(ctx) + { "trigger_type" => ctx.trigger.type.to_s } + end + end + + result = workflow_class.perform_now + + assert_equal "manual", result["trigger_type"] + end + + test "perform does not reload workflow packs" do + workflow_class = Class.new(R3x::Workflow::Base) do + def self.name + "Workflows::NoPackReload" + end + + trigger :manual + + def run(ctx) + { "trigger_type" => ctx.trigger.type.to_s } + end + end + + original_load = R3x::Workflow::PackLoader.method(:load!) + R3x::Workflow::PackLoader.singleton_class.send(:define_method, :load!) do |*| + raise "should not reload packs during workflow execution" + end + + result = workflow_class.perform_now(workflow_class.triggers.first.unique_key) + + assert_equal "manual", result["trigger_type"] + ensure + R3x::Workflow::PackLoader.singleton_class.send(:define_method, :load!, original_load) + end + test "prevents overriding perform method in subclasses" do error = assert_raises(ArgumentError) do Class.new(R3x::Workflow::Base) do From ab95496c4ace7f0a84795bda16d5be8f2b04f789 Mon Sep 17 00:00:00 2001 From: zewelor Date: Fri, 27 Mar 2026 21:29:32 +0100 Subject: [PATCH 6/7] Document workflow job caveats --- AGENTS.md | 7 +++++++ README.md | 7 +++++++ 2 files changed, 14 insertions(+) diff --git a/AGENTS.md b/AGENTS.md index b062f071..04bb9a46 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -7,6 +7,7 @@ This Rails app uses a small set of preferred libraries for common integration wo - `r3x` is a Rails API app that acts as a Ruby-native workflow executor and automation engine. - The high-level split is: framework/runtime code lives in the app and `lib/r3x/`, while user-defined workflows live under `workflows/`. - Workflows are file-based, Git-friendly, and loaded into a database-backed runtime that uses Active Job + Solid Queue for execution and recurring scheduling. +- Workflow classes are enqueued directly as Active Job classes so workflow code can use `ActiveJob::Continuable` and `step` on the real workflow job instance. - `Solid Queue` is the active job backend for app/runtime execution. Treat queueing semantics as database-backed, not Redis-backed. - In the current app configuration, `Solid Queue` is not wired through `config.solid_queue.connects_to`, so queue records use the same Active Record database connection as the app in the environments configured here. That means queue inserts can participate in the same database transaction as app writes. - If `Solid Queue` is ever moved to a separate database, or replaced with a non-database backend, revisit any code that relies on transactional integrity between app writes and job enqueueing. In that setup, `enqueue_after_transaction_commit` and related tests become important again. @@ -39,6 +40,7 @@ This Rails app uses a small set of preferred libraries for common integration wo - `R3x::ChangeDetectionJob` loads the trigger, fetches/updates `R3x::TriggerState`, and only enqueues the workflow job class itself when the trigger reports a change. - Because the app currently uses `Solid Queue` as a database-backed backend on the same Active Record database connection, code may intentionally rely on a database transaction covering both `TriggerState` updates and `perform_later`. Do not assume those guarantees survive a future backend or database split. - `R3x::RunWorkflowJob` fetches the workflow from the registry and calls `workflow_class.perform_now(trigger_key, trigger_payload: ...)` for compatibility with callers that still dispatch by workflow key. +- Known limitation: because queued workflow runs persist the concrete workflow class name, renaming or removing a workflow class across deploys can strand older queued runs with job deserialization failures. This is currently an accepted tradeoff for preserving `ActiveJob::Continuable` on the workflow job itself. - Trigger discovery is filesystem-backed through `lib/r3x/triggers/*.rb`, so trigger file names, constants, and supported types must stay aligned. ## Working with Workflows @@ -84,6 +86,11 @@ bin/workflow [options] [command] [arguments] The rake tasks exist for convenience in deployment scripts; prefer `bin/workflow` for interactive use because it has richer option parsing (e.g. `--dry-run`). +### Operational note + +- When refactoring workflow class names, remember that already queued scheduled or change-detected runs may still point at the old concrete class name. +- If a workflow class is renamed or removed, consider cleaning up pending jobs and recurring tasks created under the old class, or accept that older queued runs may fail deserialization. + ## Maintenance Warning - Keep this file synchronized with the real codebase. If you change workflow loading, trigger discovery, scheduling flow, top-level directory structure, namespaces, or the framework/user-workflow boundary, update the relevant `AGENTS.md` sections in the same change. diff --git a/README.md b/README.md index 705d4ea3..7f62eb14 100644 --- a/README.md +++ b/README.md @@ -30,3 +30,10 @@ Then open http://localhost:3000/ to view the Mission Control Jobs dashboard. ```bash bin/rails test ``` + +## Operational Notes + +- Workflow classes are enqueued directly as Active Job classes so workflows can use `ActiveJob::Continuable` and `step` on the real workflow job instance. +- Tradeoff: queued workflow runs persist the concrete workflow class name in Solid Queue. +- If a workflow class is renamed or removed before an older queued run executes, that older run may fail deserialization. +- After workflow class renames/removals, clean up pending jobs or recurring tasks that still reference the old class if you need a clean queue. From 2c4dd84f522fde659970b013475c6927f740be46 Mon Sep 17 00:00:00 2001 From: zewelor Date: Fri, 27 Mar 2026 21:37:52 +0100 Subject: [PATCH 7/7] Update Gemfile.lock dependencies - Bump action_text-trix, bigdecimal, and parser to patch versions - Update checksums in `Gemfile.lock` after dependency resolution - Result of running `bundle update` to refresh transitive gems - No application code changes; only lockfile version updates --- Gemfile.lock | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/Gemfile.lock b/Gemfile.lock index 32f09010..992efece 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -1,7 +1,7 @@ GEM remote: https://rubygems.org/ specs: - action_text-trix (2.1.17) + action_text-trix (2.1.18) railties actioncable (8.1.3) actionpack (= 8.1.3) @@ -79,7 +79,7 @@ GEM public_suffix (>= 2.0.2, < 8.0) ast (2.4.3) base64 (0.3.0) - bigdecimal (4.0.1) + bigdecimal (4.1.0) bootsnap (1.23.0) msgpack (~> 1.2) brakeman (8.0.4) @@ -318,7 +318,7 @@ GEM racc (~> 1.4) os (1.1.4) parallel (1.27.0) - parser (3.3.11.0) + parser (3.3.11.1) ast (~> 2.4.1) racc pg (1.6.3) @@ -529,7 +529,7 @@ DEPENDENCIES webmock CHECKSUMS - action_text-trix (2.1.17) sha256=b44691639d77e67169dc054ceacd1edc04d44dc3e4c6a427aa155a2beb4cc951 + action_text-trix (2.1.18) sha256=3fdb83f8bff4145d098be283cdd47ac41caf5110bfa6df4695ed7127d7fb3642 actioncable (8.1.3) sha256=e5bc7f75e44e6a22de29c4f43176927c3a9ce4824464b74ed18d8226e75a80f0 actionmailbox (8.1.3) sha256=df7da474eaa0e70df4ed5a6fef66eb3b3b0f2dbf7f14518deee8d77f1b4aae59 actionmailer (8.1.3) sha256=831f724891bb70d0aaa4d76581a6321124b6a752cb655c9346aae5479318448d @@ -544,7 +544,7 @@ CHECKSUMS addressable (2.8.9) sha256=cc154fcbe689711808a43601dee7b980238ce54368d23e127421753e46895485 ast (2.4.3) sha256=954615157c1d6a382bc27d690d973195e79db7f55e9765ac7c481c60bdb4d383 base64 (0.3.0) sha256=27337aeabad6ffae05c265c450490628ef3ebd4b67be58257393227588f5a97b - bigdecimal (4.0.1) sha256=8b07d3d065a9f921c80ceaea7c9d4ae596697295b584c296fe599dd0ad01c4a7 + bigdecimal (4.1.0) sha256=6dc07767aa3dc456ccd48e7ae70a07b474e9afd7c5bc576f80bd6da5c8dd6cae bootsnap (1.23.0) sha256=c1254f458d58558b58be0f8eb8f6eec2821456785b7cdd1e16248e2020d3f214 brakeman (8.0.4) sha256=7bf921fa9638544835df9aa7b3e720a9a72c0267f34f92135955edd80d4dcf6f builder (3.3.0) sha256=497918d2f9dca528fdca4b88d84e4ef4387256d984b8154e9d5d3fe5a9c8835f @@ -638,7 +638,7 @@ CHECKSUMS nokogiri (1.19.2-x86_64-linux-musl) sha256=93128448e61a9383a30baef041bf1f5817e22f297a1d400521e90294445069a8 os (1.1.4) sha256=57816d6a334e7bd6aed048f4b0308226c5fb027433b67d90a9ab435f35108d3f parallel (1.27.0) sha256=4ac151e1806b755fb4e2dc2332cbf0e54f2e24ba821ff2d3dcf86bf6dc4ae130 - parser (3.3.11.0) sha256=fe28ccb92d9221283ce143cb3c2e4c916e0f589ee0cc2a64867d7716354f19b1 + parser (3.3.11.1) sha256=d17ace7aabe3e72c3cc94043714be27cc6f852f104d81aa284c2281aecc65d54 pg (1.6.3) sha256=1388d0563e13d2758c1089e35e973a3249e955c659592d10e5b77c468f628a99 pg (1.6.3-aarch64-linux) sha256=0698ad563e02383c27510b76bf7d4cd2de19cd1d16a5013f375dd473e4be72ea pg (1.6.3-aarch64-linux-musl) sha256=06a75f4ea04b05140146f2a10550b8e0d9f006a79cdaf8b5b130cde40e3ecc2c