Workflows with 'fetch bucket' and multiple downstream jobs - no files in later jobs?

[greenaar]

I can't see a clear answer to this in the docs. Here's the config.

Workflow start, pull a bucket, it contains a few scripts. In the first job I run I can verify those files exist and are usable.

Job 1 passes to job 2 on success. Job has no files on startup, and no way to load them I can see. I'd expect my script loaded from the 'fetch bucket' to be there in job 2, and it's not.

Am i missing something fundamental? Is there a way to accomplish this?

In the same thread, if I write a file in job 1, it's not there in job 2 (which is run in case of success with job 1). Any way to make that happen? Note I'm not using events - the docs read like jobs would pass data automatically, and i assumed 'fetch bucket' made the contents available for all steps.

[jhuckaby]

So, the real key thing to remember here is that each job runs "in isolation", in its own unique temp directory (and commonly on different servers too). So if you want to pass data and files between jobs, you need to be explicit and tell xyOps exactly what you want to do.

Basically, if any job produces a file, and you want to pass that along to the next job, you have to tell xyOps which files you want to keep. This is done via printing a small JSON blob to STDOUT. Full instructions here: https://docs.xyops.io/#Docs/plugins/output-files

The "Fetch Bucket" action only grabs data and/or files for the current job to which it is attached. It doesn't have any knowledge of other jobs you might have in your workflow. So if you want multiple jobs to read/write from buckets, you need to attach the action to all of the job nodes.

Example:

In this example each of those two jobs fetch from a bucket when they start, and store back to a bucket when they complete.

Note that you still need to tell xyOps which files you want to store, by printing JSON to your STDOUT: https://docs.xyops.io/#Docs/plugins/output-files

Hope this helps.

[greenaar]

Ah! That's the missing link. So the first job gets the workflow's actions (including fetch/store) but downstream actions need their own. That helps a LOT.

https://docs.xyops.io/#Docs/workflows/data-passing-between-nodes (read incorrectly) could potentially use a tweak to note this behavior -- that bucket/actions are implicit on the first job being run, but need to be explicitly added for secondary jobs.

Again, thanks, this was what i needed!

[jhuckaby]

Ah, yeah, the docs are definitely still a work in progress. I'll add something in there to make it more clear. Thanks for pointing this out!