Orchestrator

[jeannekitchens]

From @rohit-joy first re-architecture component work: Set up up Workflow Orchestrator using Argo Workflows in AWS EKS, backing it with a supported DB and creating a sample C# Job to kick off Kubernetes Jobs in workflows from C# for this week starting 1/5. https://docs.google.com/document/d/1WD0AQAWuY0klJYznBST7-O93mHXSlR-k0LRQQPAqCKU/edit?tab=t.0#heading=h.28w5wj5b6qa9

[arielr-lt]

I'm looking into this

[rohit-joy]

@arielr-lt Can you please update status of this task? Do you have any questions about this?

The task is to simply set up Argo Workflows in EKS test/staging cluster with a DB backend for persistence.

Set an IP whitelist so only us authorized users can access any of the endpoints.

The Workflows will be kicked off internally through Service to Service calls, which will in turn create Kubernetes Jobs. For example, Registry can kick off a job to copy the JSON-LD to S3 and another job to index the new JSON-LDs into Elastic Search.

Manual intervention from the UI is a rare thing. The UI is simply for debugging purposes or watching the Jobs.

Once this much is done, let us know to help you set up authentication/authorization.

[arielr-lt]

@rohit-joy Argo workflows is running at https://argo.credentialengineregistry.org/. it uses an AWS RDS tiny DB for now, this environment is running with minimum resources and as a security mechanism for now I have added it a basic-auth behind a nginx ingress, please reach me out for the admin credentials when you need it.

[arielr-lt]

[rohit-joy]

Looks good! Are you posting the PR for the kubectl manifests soon?

Can you also address the IP whitelisting I mentioned above?

[arielr-lt]

Find above the PR, Credentials sent by DM

[rohit-joy]

Thank you @arielr-lt !

• The next steps are to add some dummy workflow and kicking it off using CURL or Postman.
• After that is working, create workflow to index ALL the S3 JSON-LDs into Elastic Search. And kick it off from existing Registry API.
• After that is working, create workflow to index modified S3 JSON-LDs from a Publishing operation into Elastic Search. And kick it off from Registry API calls.

Please let us know the estimates for the 3 tasks. :)

I suspect 2 hours of work for the dummy workflows, 24 hours (3-4 days) of work for the indexing workflow creation and integration with Registry? Note that the same indexing task could be used for performing full indexing versus partial indexing.

Please ensure logging is provided as part of the Workflow so that we know where it runs into problems.

[arielr-lt]

• add some dummy workflow and kicking it off using CURL or Postman. ✅