Passing data from one workflow node to subsequent bash script nodes

[matt-rgx]

Hi,

I'm building a workflow that logs its execution to a PostgreSQL database. The first node runs a shell script that inserts a row and gets back an auto-generated ID (via RETURNING id). I'd like that ID to be available in all subsequent script nodes so they can reference it in their own DB inserts.

I tested passing data through JSON output — the first node outputs: {"xy": 1, "complete": 1, "data": {"workflow_log_id": 42}}

I can see the data field is correctly parsed and stored at the workflow job level. However, in the next node's bash script, the value is not available — no INPUT_* or similar environment variable is injected.

Question: Is there a supported way to pass a value from one node's output to subsequent bash script nodes in a workflow? Or is the only option to use an external mechanism (file, env var set via another method, etc.)?

For context: the workflow has parallel branches (4 branches feeding a join node), so the ID needs to be accessible across all branches, not just the immediate next node.

Thanks

[jhuckaby]

Hi there!

So, currently the way the data input/output system is designed is to use STDIO (on both ends). So if you write some data output in one job, and wire that to a secondary job, it is passed in as part of the JSON to STDIN.

The assumption is that you are using a language that supports JSON natively, so this is easy to consume and parse. For using bash shell, you will need to use a 3rd party tool such as jq. Example of that:

#!/bin/bash

IFS= read -r line
workflow_log_id=$(printf '%s\n' "$line" | jq -r '.input.data.workflow_log_id')
echo "Workflow Log ID is: $workflow_log_id"

As for using a piece of data produced by one job across ALL jobs in the same workflow, this is more tricky. The only way to do this currently is to use the Store Bucket and Fetch Bucket actions.

On the job where you are writing the data (e.g. {"xy": 1, "complete": 1, "data": {"workflow_log_id": 42}}) attach a Store Bucket using the "On Complete" condition, like this:

Then, on all the other jobs where you need to access the data, attach a Fetch Bucket using the "On Start" condition, like this:

This is how you can share data between jobs within the same workflow, which are not "directly" wired up to each other. Just note that the bucket is global, so if you have multiple instances of the same workflow running in parallel, the fetch/store operations may step on each other.

In the future I am planning on making this easier by adding two new features:

• I was planning on introducing a new workflowData object which you can write to via JSON, and will be shared between all future workflow sub-jobs within the same run. This would work similarly to the Server User Data feature, but be stored in the current workflow, not the server.
• I was planning on passing SOME data properties in as job environment variables, but only if they are simple strings or numbers, and only if they are top-level. It gets way too crazy to try to stuff all the potentially nested data properties into env vars, when they may be in objects in other objects, in arrays, booleans, null, and so on. But for simple top-level string/number properties (like your workflow_log_id example) I can put this into an env var, in a future version of xySat.

I can't promise when these features will be added, but they are on my TODO list 😉

[matt-rgx]

Thanks a lot !

[matt-rgx]

Hi,

I’m trying to use the new workflow data/env propagation contract, but I can’t get it to behave reliably.

Setup:

• The workflow start job emits:
  {"xy":1,"complete":1,"data":{"workflow_run_id":123},"workflowData":{"workflow_run_id":123}}
• Downstream shellplug jobs are configured with json: false and pass: false
• Those jobs read:
  workflow_data_workflow_run_id or data_workflow_run_id
• The final workflow-end job uses the same env-based read

Question:
Is workflowData guaranteed to be available as workflow_data_* env vars in every downstream job launched from the
same workflow?
If yes, what is the exact required contract for this to work with shellplug jobs?
If not, what is the intended way to propagate a workflow run id across all branches and to the final end node using
the new env-based mechanism?

Also, do json: false and pass: false have any effect on whether workflowData is injected into the job
environment?

[jhuckaby]

Oh dear, you're right, there is a bug. So sorry! Brand new feature. I missed something entirely in my testing. Working on this ASAP!

[jhuckaby]

I'll answer all your questions once I get this fixed!

[jhuckaby]

Okay, all fixed! So sorry about that! I just released xySat v1.0.27, which correctly passes the workflowData properties into environment variables for jobs. There was a bug where it was trying to pull the properties from the wrong object. Oops!

Answering your questions now:

Is workflowData guaranteed to be available as workflow_data_* env vars in every downstream job launched from the same workflow?

Yup, as long as these two rules are followed:

1. The job that writes the workflowData variable is allowed to complete fully before other jobs are launched.
2. You have xySat v1.0.27+ installed on ALL of your servers (check the "Servers" tab to make sure).

If yes, what is the exact required contract for this to work with shellplug jobs?

I'm not sure what the word "contract" means in this context. Any Shell Plugin job should be able to read the variables given the two rules above (however, there was a bug that was JUST fixed today -- so sorry about that!).

Also, do json: false and pass: false have any effect on whether workflowData is injected into the job environment?

Nope, and also side-note: json was removed from the Shell Plugin a few versions ago. So you can edit the Plugin and remove that checkbox if you want. It's unused. (It always interprets JSON now).

The pass flag just means that the Shell Plugin will echo all the input data as output (it doesn't affect workflowData).

Thank you for reporting this issue!

[matt-rgx]

I’m seeing a regression around workflowData propagation in a workflow fan-out after a container redeploy/recreate.

Environment

• xyOps container: ghcr.io/pixlcore/xyops:latest
• Image digest: sha256:4abd4e59623e0e7159d840081ee06630736598f60d185695a47e238
  60ae5a6e8
• Version label: v1.0.51

What changed

• No workflow definition changes between the working run and the failing run.
• The infrastructure stack was redeployed/recreated by Forgejo, but xyOps came back on the same image digest.
• There were no missing container env vars on xyOps.

Observed behavior

• The workflow start node correctly emits both:
  • input.data.workflow_run_id
  • workflowData.workflow_run_id
• On the very next fan-out level, child jobs still receive
  input.data.workflow_run_id, but workflowData becomes null.
• On the previous run, the exact same workflow propagated workflowData correctly to those same child jobs.

Concrete example

• Working run the day before: children had
  workflowData.workflow_run_id=10.
• Failing run after redeploy: start node had workflowData.workflow_run_id=11, but children got
  workflowData=null immediately after fan-out.

Impact

• Our workflow_end step depends on workflow_data_workflow_run_id.
• Because workflowData is lost, the end node cannot close the workflow run in our SQL log table.
• The workflow itself completes, but the workflow_logs row stays running.

Question

• Is workflowData propagation across workflow fan-out expected to survive a redeploy/recreate of the xyOps container?
• Is there a known regression in v1.0.51 around workflowData or the workflow split controller / job cloning path?
• If there is a specific fix in a newer release, I’d like to
  verify whether it resolves this exact issue.

[matt-rgx]

I’m seeing a regression around workflowData propagation in a workflow fan-out after a container redeploy/recreate.

Environment

• xyOps container: ghcr.io/pixlcore/xyops:latest
• Image digest: sha256:4abd4e59623e0e7159d840081ee06630736598f60d185695a47e238
  60ae5a6e8
• Version label: v1.0.51

What changed

• No workflow definition changes between the working run and the failing run.
• The infrastructure stack was redeployed/recreated by Forgejo, but xyOps came back on the same image digest.
• There were no missing container env vars on xyOps.

Observed behavior

• The workflow start node correctly emits both:
  • input.data.workflow_run_id
  • workflowData.workflow_run_id
• On the very next fan-out level, child jobs still receive
  input.data.workflow_run_id, but workflowData becomes null.
• On the previous run, the exact same workflow propagated workflowData correctly to those same child jobs.

Concrete example

• Working run the day before: children had
  workflowData.workflow_run_id=10.
• Failing run after redeploy: start node had workflowData.workflow_run_id=11, but children got
  workflowData=null immediately after fan-out.

Impact

• Our workflow_end step depends on workflow_data_workflow_run_id.
• Because workflowData is lost, the end node cannot close the workflow run in our SQL log table.
• The workflow itself completes, but the workflow_logs row stays running.

Question

• Is workflowData propagation across workflow fan-out expected to survive a redeploy/recreate of the xyOps container?
• Is there a known regression in v1.0.51 around workflowData or the workflow split controller / job cloning path?
• If there is a specific fix in a newer release, I’d like to
  verify whether it resolves this exact issue.

[jhuckaby]

It sounds like you have a race condition, where the job that writes the data into workflowData doesn't complete fully before the other jobs start. Jobs that run in parallel can sometimes change order.

workflowData is only "committed" (written to shared storage) when the job that writes it FULLY completes.

Can you post a screenshot of your workflow map?

[matt-rgx]

The workflow is organized so that workflow start is the only node writing workflowData / workflow_run_id.
All downstream branches are configured with on complete, so they only start after the start node has fully finished.
So this does not look like a race between the writer and the consumers. I can’t provide a screenshot right now, but I can describe the graph if needed.

[jhuckaby]

Okay, so if your workflowData writing job completes fully before the other jobs run, then the only other thing it could possibly be is that some of your servers have an older xySat (xyOps Satellite) version. They all need xySat v1.0.27. Anything older on ANY of your servers will cause the workflowData to not propagate to jobs. The workflowData feature is brand new, and requires the very latest xyOps (v1.0.50+) and xySat (v1.0.27+) versions across your entire fleet.

Check the "Servers" tab to make sure all your servers are up to date. Also, if you have xyOps v1.0.52+ it will visually indicate which servers are up to date using a green checkbox.

Is workflowData propagation across workflow fan-out expected to survive a redeploy/recreate of the xyOps container?

Are you talking about doing a redeploy / recreate of the xyOps container WHILE the workflow is actively running? I am having a bit of trouble understanding this question.

xyOps should automatically recover and resume active workflows in progress if your primary conductor server is "restarted", but for a "redeploy / recreate" of the container to work seamlessly while workflows are actively running, you will need a full multi-conductor cluster, with auto-failover.

Is there a known regression in v1.0.51 around workflowData or the workflow split controller / job cloning path?

No. There have been no changes to the workflowData feature after the initial addition in xyOps v1.0.50. However, there HAVE been major bug fixes in xySat (xyOps Satellite) regarding this feature, and you need xySat v1.0.27 across ALL of your servers for the workflowData feature to work correctly.

[matt-rgx]

No. There have been no changes to the workflowData feature after the initial addition in xyOps v1.0.50. However, there HAVE been major bug fixes in xySat (xyOps Satellite) regarding this feature, and you need xySat v1.0.27 across ALL of your servers for the workflowData feature to work correctly.

Actually, i've found that i was in 1.0.23 on my server. But the really weird thing is that i'm sure like 99.99999% that i've already upgrade to 1.0.27 last week. And the workflowData worked for about 2 days back then. So i don't understand how i'm back to 1.0.23 now.

EDIT: I think I found the missing piece.

I re-upgraded the local satellite via the xyOps UI, confirmed it was on xysat 1.0.28, and then tested a Docker
recreate of the xyops container.

Right after the recreate, the local satellite had reverted back to xysat 1.0.23.

So that seems to explain the confusing behavior on my side:

• I really did upgrade it before
• workflowData worked for a while
• then a later container recreate likely reset the embedded local satellite back to 1.0.23
• and after that, workflowData propagation started failing again

So my current suspicion is that the UI upgrade of the local embedded satellite does not survive a container recreate.