Production rollback error from existing tag

[dallas-fletchall]

What product are you having troubles with?

Console

What Console version are you using?

v14.4.0

Description

During production rollback, a user experienced an error. After investigation, it seems identical to issue #634:

The original issue contents:

When trying to rollback a production deployment, We receive an error, but the subsequent commit to GitHub still succeeds.

When looking at logs, we see that the error has to do with creation of git tag, which already exists.

This seems to only be an issue on production environments.

We had verified that the fix worked on a previous version and are wondering if a possible regression has happened.

Actual Outcome

Error in the deploy service after production rollback:

...
  "level": 50,
  "hostname": "deploy-service-647ccbb6c9-rd2b7",
  "reqId": "a799e870-fcb8-423b-9496-b4741aedfa32",
  "projectId": "65e0bc125c723045cb345985",
  "refInfo": {
    "name": "2.22.10",
    "type": "version"
  },
  "envId": "prod",
  "message": "Reference already exists - https://docs.github.com/rest/git/refs#create-a-reference",
  "error": {
    "type": "HTTPError",
    "message": "Response code 500 (Internal Server Error)",
    "stack": "HTTPError: Response code 500 (Internal Server Error)\n    at Request.<anonymous> (/home/node/app/node_modules/got/dist/source/as-promise/index.js:118:42)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)",
    "name": "HTTPError",
    "code": "ERR_NON_2XX_3XX_RESPONSE",
...

The tag also exists in GitHub like before. The commit worked fine, so not in sync with the console for the latest deployed version.

Expected Outcome

We should be able to rollback to a previous version that has already been deployed. The version in Mia should match the version deployed.

[ThisIsDemetrio]

Hi @dallas-fletchall . Apologies for give it back to you so late.

There are no updates on this part of the Console in the last versions. The issue in #634 has been solved after releasing the 14.1.0 version of the Console (chart v15.0.0) and since then there are not been updates on deployments of GitHub-based project, nor on GitHub-based repository management.

If you still encounter this issue, would you mind sharing more details? As example, did you see in the logs if there are additional error message related to the same reqId? I imagine you're using GitHub to store the project repository and you're using it as well to deploy the project, but are there any tags created manually or modified in some sort? Have you by chance found this issue also with other tags?

By the way, on thursday the new LTS version of the Console will be released (14.4.2 of the Console - you can already read the release notes here - the chart version will be updated in a few days), so you might opt to install that chart and see if the issue solves. If that happens, we will investigate also the previous chart to ensure it wasn't released with missing parts.

D

[dallas-fletchall]

Mia-Platform v14.5.1 is working as intended. Thank you!

[dallas-fletchall]

Actually, we are still seeing the issue, but not on every Project. Currently, the project that I have tested and the project that is failing are in different Companies.

Could there be differences in the Company and/or Project that could be the cause? Is there a race condition that could be happening?

The best information that I have currently is that a "working" project will create tags in GitHub like X.Y.Z-prod-{0-9}[10], but projects that fail will not have any tags with the trailing -{0-9}[10]. What other information would be helpful to solve this issue.

[ThisIsDemetrio]

Hi @dallas-fletchall

Thanks for the update. So sorry to hear that you still encountering issues on this. We're going to take some time to inverstigate it thoroughly, but as fasr as I know, there are no differences or particular logic that differ from company to company (or from project to project) on the deployment of projects and generation of tags/commit with providers.

We will try to replicate the issue creating some edge cases on our test environments. Let's keep each other updated if we find out something new.

Cheers,
|D|