Introduction

This repository, gha_mlops_demo, demonstrates two methods for artifact migration (e.g. models, environments, assets) between workspaces in Azure Machine Learning (Azure ML) using GitHub Actions as the orchestrator:

1. Using Model Registries
2. Using Docker Containers

The goal of this project is to showcase how to efficiently manage and automate the transfer of machine learning models and other artifacts across different Azure ML workspaces, leveraging the power of GitHub Actions for continuous integration and continuous deployment (CI/CD) for non-standard ML Models. In this demo Meta's "Prophet" forecasting model is extantiated in python scripts within the src directly to emulate single-stage/one-shot ML model. Additionally, the prophet sample dataset, example_retail_sales.csv is used as the input and is stored within an ADLS Gen 2 store. Additionally, model outputs are written to an ADLS Gen 2 store. External stores are used in this demo to show how to interface with external stores in MLOps workflows.

By following the examples and workflows provided in this repository, you will learn how to:

1. Set up GitHub Actions workflows to automate the migration of artifacts between Azure ML workspaces.
2. Use Azure CLI commands within GitHub Actions to manage and transfer artifacts securely and efficiently.
3. Implement best practices for CI/CD in machine learning projects, ensuring smooth and reliable deployment pipelines.

This demo is designed to help data scientists, machine learning engineers, and DevOps practitioners understand and implement robust artifact migration strategies in their Azure ML projects.

Project Structure

• .github/workflows/
  Contains GitHub Actions workflows. These files automate tasks such as model/environment registration, running jobs in different Azure ML workspaces, and promoting assets to central registries.

• data/
  Stores the sample dataset (example_retail_sales.csv) used as input for the forecasting model.

• docs/assets/
  Contains visual diagrams that illustrate both the Model Registry and Container-based architectures, along with screenshots to assist in setting up GitHub secrets.

• jobs/
  Houses job definitions for Azure ML. These YAML files detail how to run forecasting jobs using different assets (model or container) and how to create a centralized model registry.

• src/
  Contains the Python source code for the forecasting model. The forecast_one_shot.py script implements the forecasting procedure using Facebook’s Prophet library.

• Dockerfile
  Defines the steps to build a Docker image that encapsulates the forecasting script and its dependencies. This image is used to create a portable Azure ML Environment.

• conda.yaml & requirements.txt
  List all the dependencies required by the forecast model. The conda.yaml file is used for registering an environment in Azure ML, while requirements.txt is used during Docker image construction.

• .dockerignore & .gitignore
  Define patterns for files and directories to be excluded from the Docker build context and version control tracking respectively, ensuring a clean build and repository state.

Architecture Description

Method 1: Migration using Model Registries

This architecture, leverages Azure ML Model Registries to migrate artificats between workspaces. By default, Azure ML Workspaces include a registry within a workspace-dedicated storage account to manage artifacts. However, when migrating to other workspaces we'll need to use a global registry as the workspace specific registries are only accesible within it's associated Azure ML workspace.

In the diagram below:

1. The model source code, environment, scripts, and CI/CD Github actions + associated Azure ML Jobs are stored in a GitHub repository. When changes to the model files are committed a GitHub action is launched to package + register the model within the Azure ML Dev workspace. Note, this will only be available within the dev workspace. Likewise when changes to the environment are committed (e.g. env/conda.yaml), the new environment will pushed to the workspace registry

2. Test the model by running it on compute within the Azure ML Workspace. The model and environment will be loaded from the Workspace registry. You can optionally interact with external storage accounts that are registered (linked) to the workspace as is done for this demo.

3. After testing the model, if it's ready for production you can promote the model and/or environment, depending on what was updated, to a central repository thats accessible from multiple workspaces.

4. Run the model in production. The model and environment will be pulled from the central repository (see prev. step) and ran on compute within the production workspace.

Method 2: Migration using Containers

This architecture, leverages Docker containers to migrate artificats between workspaces. In this case all artificates (environment, model, scripts etc.) are contained with a single portable Docker container. The Docker container may be stored in workspace/central registries as an "environment" as is done in Method 1 OR in the Azure Container Registry (ACR) service. By using container, the model may be ran outside of Azure ML or even external to Azure.

In the diagram below:

1. The model source code, environment, scripts, and CI/CD Github actions + associated Azure ML Jobs are stored in a GitHub repository. When changes to the model or environment files are committed a GitHub action is launched to package + register the Docker image within the Azure ML Dev workspace. Note, this will only be available within the dev workspace.

2. Test the model by running it on compute within the Azure ML Dev Workspace. The model and environment will be both loaded from the Docker image stored in the workspace environment registry. You can optionally interact with external storage accounts that are registered (linked) to the workspace as is done for this demo.

3. After testing the model, if it's ready for production you can promote the model and environment to a central repository thats accessible from multiple workspaces. In this demo ACR is used, but a central repository such as what was used in Method #1 may be used as well.

4. Run the model in production. The Docker image will be pulled from ACR (see prev. step) and ran on compute within the production workspace.

Getting Started

Prerequisites

• Azure subscription
• At least 2 Azure Machine Learning workspaces
• GitHub account w/ access to GitHub Actions
• Azure CLI setup and configured on your local machine (docs)

Setup

1. Commit the contents of this repository to a new GitHub repository associated with your organization

2. Register a new application service-principal for your GitHub actions. GitHubActionsSp is the service principal used in this demo, we use the same service principal for all our workflows, however you configured multiple service principals to different workflows if you choose.

   az ad sp create-for-rbac --name "GitHubActionsSP" --role contributor --scopes /subscriptions/<subscription_num> --json-auth
   
   # sample output:
   {
     "clientId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
     "clientSecret": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
     "subscriptionId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
     "tenantId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
     "activeDirectoryEndpointUrl": "https://login.microsoftonline.com",
     "resourceManagerEndpointUrl": "https://management.azure.com/",
     "activeDirectoryGraphResourceId": "https://graph.windows.net/",
     "sqlManagementEndpointUrl": "https://management.core.windows.net:8443/",
     "galleryEndpointUrl": "https://gallery.azure.com/",
     "managementEndpointUrl": "https://management.core.windows.net/"
   }
   
   

   Store the JSON object as it will be needed in the subsequent steps.

3. In your GitHub repository navigate to <repository> > Settings -> Security -> Secrets and variables -> Actions. Press the New repository secret button, enter AZURE_CREDENTIALS in the name field and past the contents of the Service Principal JSON from step #2 in the secret value and press Add secret. Repeat this process for each of the below secrets.

   Secret Name	Description
   AML_DEV_SUBSCRIPTION	Azure ML Dev Workspace subscription
   AML_DEV_RESOURCE_GROUP	Azure ML Dev Workspace Resource Group
   AML_DEV_WORKSPACE	Azure ML Dev Workspace name
   AML_PROD_SUBSCRIPTION	Azure ML Production Workspace subscription
   AML_PROD_RESOURCE_GROUP	Azure ML Production Workspace Resource Group
   AML_PROD_WORKSPACE	Azure ML Production Workspace name

   

4. Next, compute will be created in the dev and prod Azure ML workspaces. For this demo, a CPU cluster with min-instances size of 0 is used. By setting the minimum instance size to 0, after the amount of time specified in idle-time-before-scale-dow, the cluster nodes will be spun-down. Feel free to change max-instances to a greater number then used in this demo (1). The size parameter specified the compute instance used (CPU/GPU, RAM, Storage etc.), see (docs) for more info.

Run the below command for twice, one for each workspace and note the service princiapl ID (principalId) returned.

az ml compute create --name cpu-cluster \
  --type AmlCompute \
  --size Standard_D2_v2 \
  --min-instances 0 \
  --max-instances 1 \
  --idle-time-before-scale-down 60 \
  --identity-type SystemAssigned \
  --subscription <dev/prod subscription> \ 
  --resource-group <dev/prod resource group> \
  --workspace-name <dev/prod workspace name>

5. In this demo an ADLS Gen 2 store is used to store the input data and the output forecast results. If you haven't already, create at least one ADLS Gen 2 store. For each store run the following command to retreive the "objectId" which will be used in the next steps.

az storage account show --name <storage-account-name> --resource-group <resource-group-name> --query id -o tsv

# sample output:
/subscriptions/<subscription-id>/resourceGroups/<resource-group-name>/providers/Microsoft.Storage/storageAccounts/<storage-account-name>

6. Grant the Azure ML Prod & Dev compute clusters access to the external data store(s). Run the command below for each workspace and for each ADLS Gen 2 store where ADLS_ID were the ID's of the ADLS Gen 2 stores retrieved in step #5 and the COMPUTE_PRINCIPAL_ID was from step #4.

az role assignment create \
  --assignee <COMPUTE_PRINCIPAL_ID> \
  --role "Storage Blob Data Contributor" \
  --scope <ADLS_ID>

Model Repository Setup

To use the model registry artificat migration, a centralized artifact repository needs to be created outside of our Azure Workspaces to store artificats include the model and environment. Please folow the below steps:

1. Edit the jobs/create_model_registry.yaml: specify the regions you would like the model regsitry to be accesible from as well as the name. Be sure to include the regions for your Azure ML workspaces. Additonal configuration is descirbed in Manage Azure Machine Learning Registries

2. Save the .yaml from step #1. Run the following command in the root directory of this repository. RESOURCE_GROUP specifies the RG that will contain the registry and does not need to be one of the Azure ML RG's.

az ml registry create --resource-group <RESOURCE_GROUP> --file jobs/create_model_registry.yml

Azure Containter Setup

To migrate artifacte between workspaces using containers, the containers are stored in the Azure Container Registry (ACR) which may be accessed not-only in different Azure ML workspaces but even in VM or external compute. Follow the steps below to setup ACR:

1. Create a new registry in ACR by running the following command. RESOURCE_GROUP specifies the RG that will contain the registry and does not need to be one of the Azure ML RG's.

az acr create \
  --name <ACR_NAME> \
  --resource-group <RESOURCE_GROUP> \
  --sku Basic \
  --admin-enabled true

In the response save the id field as it will be needed to link to our Azure ML workspaces:

{
  "adminUserEnabled": false,
  "creationDate": "2019-01-08T22:32:13.175925+00:00",
  "id": "/subscriptions/aaaa0a0a-bb1b-cc2c-dd3d-eeeeee4e4e4e/resourceGroups/myResourceGroup/providers/Microsoft.ContainerRegistry/registries/mycontainerregistry",
  "location": "eastus",
  "loginServer": "mycontainerregistry.azurecr.io",
  "name": "mycontainerregistry",
  "provisioningState": "Succeeded",
  "resourceGroup": "myResourceGroup",
  "sku": {
    "name": "Basic",
    "tier": "Basic"
  },
  "status": null,
  "storageAccount": null,
  "tags": {},
  "type": "Microsoft.ContainerRegistry/registries"
}

2. Run the following command to verify connection to the ACR:

az acr login -n <ACR_NAME> --expose-token

# sample response
{
  "accessToken": "Hx1QTFyNwfbVHJS6vBfLemxRDuHYiEKHjE8CjENSQTFQTGdWU6f8mWDN2gLC6LtBmTCe824aaQkbwb6RLCS9eJ3ZdyTtKiMkdZxaejwNPWYnt3RHiYmEuZHxRPku8pm1SLM93PS5cBwPmdwUP1qv8wMSK2hm2f7SvCxp80Fv2rUftwA4dNhP0ZxuyF2gTjRN8WgPfdyapT6FmzpM2CR37cZSw2v00QgnSGbt8SuA7tBP7neFDLUP3t2XZ0ght6C09LKtYR7VRaazS2JTjJfcQdkBM0y7mEd1H1n3iC9twatZky6QkBwYj9jQu7YeziJReq2VEEGYADifkn50nhD5KpRaBpgu6SycaGmraAepiUt3gvbtnQYKjMDtBtUhYPJFn5rWq10f5SZXE5JqPa6xjqa8qX8DRQ7UNkqZfYmJHyP5Lu1F0kk3BWRxYpP6puW5pT5ZSLqPXCSmLv5zLvL1yvUrJeG1mJ6VCZX8NDMuAhEVe0iwKDzjDbxuAeRXEGNhmcHGhA17xrjiKhzLzxwM0jfrmvuaY7BmG963udCmP1unzJU4DwE1wdA4FxY7rbR5PWt2fy9NNzTTJzY3HuNq62SjQXt2jAYgy1RtJANmwxLAkwJKypSi54xpN6L5DNWriPF0gaj3utDefd3Ma7T4fSrGHHhveqdaB37KD70eWa4R8ri9w5kyrwrUtTYyUTnh0rA4Z2hTmvZ5aPr7Wuvw5H7P7thujwqNQURKtJAiM2GpKB1bfmhVgbNQQLwym5QFF6ExWFGAmfLHprEa5HJ99NS8Pd1XAVJTPLriKNLH3c80DJSybNdvKchjumY3B5GSAdi0Pj8Hq0PNgwae38ZUKbn65dk93XcFxEUyWZX962YVBH2WLFBFL4hL9mCq8Gn1hzYYt0ub2FJaCv7bN7jKCECghuSZHueN8b6KTwmGfpKQ4Kx4xMXmYzWjLbubtvv6TbJ8PWPGBSJ1yM1ER2rRzMyPhVetyrC7VWBxvuP9",
  "loginServer": "*********"
}

3. Associate the ACR registry to the Azure ML Prod workspace. This allows access to the images store din the ACR. Optionally repeat the command for the dev workspace if you plan on using ACR images in that workspace as well. ACR_ID is the ID from step #1.

az ml workspace update \
  --name <AML_WORKSPACE_NAME> \
  --resource-group <AML_RESOURCE_GROUP> \
  --container-registry "<ACR_ID>"

3. Get the Service Principal ID's of your compute:

az ml compute list \
  --workspace-name <AML_WORKSPACE_NAME> \
  --resource-group <AML_RESOURCE_GROUP> \
  --query "[].{Name:name, PrincipalId:identity.principal_id}" \
  --output table

# sample result:
Name         PrincipalId
-----------  ------------------------------------
cpu-cluster  ********-****-****-****-********

4. Grant compute read access to the ACR.

az role assignment create \
  --assignee <COMPUTE_PRINCIPAL_ID> \
  --role "AcrPull" \
  --scope <ACR_ID>

Performing Artificat Migration

Navigate to your GitHub repo -> Actions and note the available actions. The actions that start with "C" denote container migration flow and the actions that start with "M" denote the model registry migration flow.

Method 1: Migration using Model Registries

1. M1 - Register Environment in Dev

   • Upon the changes to the environment, this action automatically pushes the updated environment to the Azure ML dev workspace registyr. In this demo our environment is a conda.yaml file that includes the enviroment to run our ML Model which includes all the python packages in requirements.txt
   • Next the script determines what the next version of the environment should be: If the environment has not been registered yet the version will be "1". If it has, the latest version will be incremented
   • Finally the environment is uploaded to the workspace registry

2. M1 - Register Model in Dev

   • Upon the changes to the model (files in "src/"), this action automatically pushes the updated model files to the Azure ML dev model registry.
   • Next the script determines what the next version of the model should be: If the model has not been registered yet the version will be "1". If it has, the latest version will be incremented
   • Finally the model is uploaded to the workspace registry

3. M2 - Run Model in Dev

   • Submit a job in the Azure ML DEV workspace to run the model. The jobs/run_forecast_model.yaml file specifies the job including: input/output path from the ADLS Gen 2 stores, the environment, the model to use and the compute to use. Since registered models in the workspace require an explicit version we first retrieve the latest model version and pass it into our job yaml when we sumit the job.

4. M3 - Promote Model & Env to Central Repository

   • Get the latest version of the model & enviromnet in the Azure ML Dev workspace
   • Get the latest versions of the model & enviromnent in the central repository
   • If the workspaces have a newer version of the model or repository, upload them to the central repository

5. M4 - Run Model in Prod

   • Submit a job in the Azure ML PROD workspace to run the model. The jobs/run_forecast_model.yaml file specifies the job including: input/output path from the ADLS Gen 2 stores, the environment, the model to use and the compute to use. Paths for the model/env in the central repository are passed into the YAML job file overwriting any existing values

Method 2: Migration using Containers

1. C1 - Register Container as Azure ML Environment

   • Environments in Azure ML can be docker images including both standard community docker images as well as custom docker images. For the custom case, a Dockerfile and build context (input files) are submitted the Azure ML environment. Azure ML then builds and stores the Docker image. In this demo we use this ability to register our Docker container wihtin the dev workspace for confined testing.
   • Similar to action M1 the environment workspace registry is checked to see if this container has been registered before. If it has, the version is incremented. If it has not, the verion is set to 1.
   • In this demo the Docker container is BOTH the model and enviroment.

2. C2 - Promote Container to Central Registry (ACR)

   • This is similar to action M3 where we push our model/environment to a central repository accessible from multiple workspaces. In this demo Azure Container Registry (ACR) is used, but storing the container in a central model registry as an environment may be used as well

3. C3 - Run container in Dev

   • Submit a job in the Azure ML DEV workspace to run the model. The jobs/run_forecast_container.yaml file specifies the job including: input/output path from the ADLS Gen 2 stores, the environment and the compute to use. This is similar to the model registry job except our the Docker container contains both the model AND environmnet so we do not need to explicity specify a model entry point.

4. C4 - Run container in Dev

   • Submit a job in the Azure ML DEV workspace to run the model. The jobs/run_forecast_container.yaml file specifies the job including: input/output path from the ADLS Gen 2 stores, the environment and the compute to use. This is similar to C3 except the Docker container environment is pulled from ACR instead of the workspace registry

Future Work

• Explore additional methods for artifact migration.
• Implement more complex CI/CD pipelines and combined workflows
• Add model/environment pruning actions