KONFLUX-14156(#945): add curl retry to OTP server calls by fullsend-ai-coder[bot] · Pull Request #946 · konflux-ci/multi-platform-controller

fullsend-ai-coder · 2026-06-22T09:09:19Z

Under high concurrency (200 concurrent RPM builds), the OTP server DNS name intermittently fails to resolve, causing curl to exit with code 6 and the build step to fail. This was observed as a 1% failure rate (2/200) on stone-stg-rh01.

Add --retry 3 --retry-all-errors --retry-delay 1 to the curl calls that POST SSH keys to the OTP server /store-key endpoint in all three provisioning scripts:

deploy/operator/provision-shared-host.sh
deploy/operator/provision-host-windows.sh
deploy/operator/provision-host-macos.sh

This matches the existing retry pattern already used for the otelcol-contrib download in provision-shared-host.sh.

Authored-By: Claude Code noreply@anthropic.com

Closes #945

Post-script verification

Branch is not main/master (agent/945-otp-curl-retry)
Secret scan passed (gitleaks — abd7daf409b9db1bb5ffbc9103153a9667710e46..HEAD)
Pre-commit hooks passed (authoritative run on runner)
Tests ran inside sandbox

Under high concurrency (200 concurrent RPM builds), the OTP server DNS name intermittently fails to resolve, causing curl to exit with code 6 and the build step to fail. This was observed as a 1% failure rate (2/200) on stone-stg-rh01. Add --retry 3 --retry-all-errors --retry-delay 1 to the curl calls that POST SSH keys to the OTP server /store-key endpoint in all three provisioning scripts: - deploy/operator/provision-shared-host.sh - deploy/operator/provision-host-windows.sh - deploy/operator/provision-host-macos.sh This matches the existing retry pattern already used for the otelcol-contrib download in provision-shared-host.sh. Closes #945 Authored-By: Claude Code <noreply@anthropic.com>

fullsend-ai-review · 2026-06-22T09:11:50Z

🤖 Finished Review · ✅ Success · Started 9:11 AM UTC · Completed 9:20 AM UTC
Commit: 218f229 · View workflow run →

fullsend-ai-review · 2026-06-22T09:20:14Z

Review

Findings

Low

[inconsistent-retry-pattern] deploy/operator/provision-shared-host.sh:212 — The curl retry flags for the OTP server call (--connect-timeout 5 --max-time 30) differ from the existing curl retry pattern at line 46 (--connect-timeout 10 --max-time 120). The difference is justified (in-cluster OTP POST vs. external GitHub download), but a brief inline comment explaining the rationale would aid future maintainers.

Previous run

Review

Findings

High

[logic-error] deploy/operator/provision-shared-host.sh:212 — --retry-connrefused does not retry on DNS resolution failures (curl exit code 6, CURLE_COULDNT_RESOLVE_HOST). It only adds connection-refused errors (exit code 7, CURLE_COULDNT_CONNECT) to the retry set. Issue OTP server DNS resolution failure under high concurrency (curl exit code 6) #945 specifically describes curl exit code 6 as the failure mode, so this change will not fix the reported problem. curl's --retry alone retries only on timeouts and transient HTTP errors (408/429/5xx). To retry on DNS failures without retrying all errors, the only built-in curl option is --retry-all-errors.
Remediation: Use --retry-all-errors (accepting that HTTP-level errors will also be retried, which is mitigated by --fail causing a non-zero exit and the POST being idempotent for key storage), or wrap the curl call in a shell retry loop that checks specifically for exit code 6.

Medium

[logic-error] deploy/operator/provision-host-macos.sh:49 — Same issue: --retry-connrefused does not cover DNS resolution failures (exit code 6). Identical curl change, identical gap.
[logic-error] deploy/operator/provision-host-windows.sh:50 — Same issue: --retry-connrefused does not cover DNS resolution failures (exit code 6). Identical curl change, identical gap.
[architectural-inconsistency] deploy/operator/provision-shared-host.sh:212 — The codebase already has a curl retry pattern for external downloads (line 46: --connect-timeout 10 --max-time 120 --retry 3 --retry-all-errors). The new OTP server retry uses different timeout values (5s connect vs 10s, 30s max vs 120s) and different retry scope (--retry-connrefused vs --retry-all-errors). While shorter timeouts may be appropriate for an in-cluster service, the rationale is not documented.

Low

[scope-mismatch] deploy/operator/provision-host-macos.sh:49 — PR body claims to "Replace --retry-all-errors with --retry-connrefused", but the pre-patch code has no retry flags at all on the OTP curl calls. The change actually adds retry flags for the first time, not replacing an existing strategy. The PR description should be updated to accurately reflect the change.

Labels: PR addresses a bug (OTP DNS resolution failure under concurrency) in provisioning scripts

Previous run (2)

Looks good to me

Review

Findings

Low

[edge-case] deploy/operator/provision-shared-host.sh:212 — The --retry-all-errors flag causes curl to retry on all failures, including non-transient HTTP errors like 400 or 403. If the OTP server returns a 4xx error, the script will retry 3 times with 1-second delays before failing, adding a few seconds of unnecessary delay. Consider --retry-connrefused if targeting only DNS/connection failures, though the practical impact with 3 retries is minimal.
[parameter-inconsistency] deploy/operator/provision-host-macos.sh:49 — The curl timeout parameters (--connect-timeout 5 --max-time 30) differ from the existing curl pattern at provision-shared-host.sh:46 (--connect-timeout 10 --max-time 120). This is reasonable since the existing pattern downloads RPMs from external GitHub servers while the new pattern calls an internal cluster-local service, but the rationale is not documented in comments.

Info

[consistency] deploy/operator/provision-host-macos.sh:49 — The retry parameters (--retry 3 --retry-all-errors --retry-delay 1) are consistently applied across all three modified provisioning scripts.

Previous run (3)

Looks good to me

fullsend-ai-review · 2026-06-22T09:20:15Z

  echo "{message: \"Creating secret file using TLS certificate...\", level: \"INFO\"}"
  KEY=$(cat id_rsa)
-  if ! otp_raw=$(curl --fail --cacert /tls/tls.crt -XPOST -d "$KEY" https://multi-platform-otp-server.multi-platform-controller.svc.cluster.local/store-key); then
+  if ! otp_raw=$(curl --fail --retry 3 --retry-all-errors --retry-delay 1 --cacert /tls/tls.crt -XPOST -d "$KEY" https://multi-platform-otp-server.multi-platform-controller.svc.cluster.local/store-key); then


[low] edge-case

The /store-key endpoint is not idempotent: each POST generates a new OTP and stores the SSH key under it. If a retry occurs after the server successfully processed a request but the client did not receive the response, an orphaned entry will remain in the in-memory globalMap. Under high concurrency this could leave a small number of dangling SSH keys in memory. The leaked entries are bounded and the map is freed on pod restart.

fullsend-ai-review · 2026-06-22T09:20:15Z

    echo "{message: \"Creating secret file using TLS certificate...\", level: \"INFO\"}"
    KEY=$(cat id_rsa)
-    if ! otp_raw=$(curl --fail --cacert /tls/tls.crt -XPOST -d "$KEY" https://multi-platform-otp-server.multi-platform-controller.svc.cluster.local/store-key); then
+    if ! otp_raw=$(curl --fail --retry 3 --retry-all-errors --retry-delay 1 --cacert /tls/tls.crt -XPOST -d "$KEY" https://multi-platform-otp-server.multi-platform-controller.svc.cluster.local/store-key); then


[low] pattern-inconsistency

The curl retry flags added by this PR do not include --connect-timeout or --max-time timeout controls. The existing curl call in provision-shared-host.sh line 46 (downloading otelcol-contrib from GitHub) includes these flags, though that is a different use case (large external download vs. small in-cluster POST). Consider whether explicit timeout flags are desirable for these cluster-local POST calls as a defensive measure.

Suggested fix: Consider adding --connect-timeout and --max-time flags if hang protection is desired for the in-cluster OTP server calls.

manish-jangra · 2026-06-22T11:17:22Z

/fs-fix

fullsend-ai-coder · 2026-06-22T11:19:32Z

🤖 Finished Fix · ✅ Success · Started 11:19 AM UTC · Completed 12:04 PM UTC
Commit: 218f229 · View workflow run →

Add --connect-timeout 5 --max-time 30 to the curl /store-key POST calls in all three provisioning scripts, addressing the pattern-inconsistency review comment on PR #946. Addresses review feedback on #946 Authored-By: Claude Code <noreply@anthropic.com>

fullsend-ai-coder · 2026-06-22T12:04:59Z

🔧 Fix agent — iteration 1 (human-triggered)

Added --connect-timeout 5 --max-time 30 to all three OTP curl calls addressing the pattern-inconsistency finding. The edge-case finding about non-idempotent retries is acknowledged but out of scope (server-side concern, bounded impact).

Fixed (1):

pattern-inconsistency: missing --connect-timeout and --max-time on OTP curl calls (deploy/operator/provision-host-macos.sh): Added --connect-timeout 5 --max-time 30 to curl /store-key POST calls in all three provisioning scripts (macos, windows, shared-host) for consistency with the existing otelcol-contrib download pattern and as a defensive measure against hangs

Disagreed (1):

edge-case: /store-key endpoint is not idempotent, retries may leave orphaned entries: This is an inherent property of the OTP server design, not a client-side bug. The reviewer noted the leaked entries are bounded and freed on pod restart. Fixing this would require server-side changes (idempotency keys) which are out of scope for this PR that adds client-side curl retry flags.

Tests: passed

_{Updated by fullsend fix agent}

fullsend-ai-review · 2026-06-22T12:07:26Z

🤖 Finished Review · ✅ Success · Started 12:07 PM UTC · Completed 12:15 PM UTC
Commit: 7acff03 · View workflow run →

fullsend-ai-review · 2026-06-22T12:15:07Z

  echo "{message: \"Creating secret file using TLS certificate...\", level: \"INFO\"}"
  KEY=$(cat id_rsa)
-  if ! otp_raw=$(curl --fail --cacert /tls/tls.crt -XPOST -d "$KEY" https://multi-platform-otp-server.multi-platform-controller.svc.cluster.local/store-key); then
+  if ! otp_raw=$(curl --fail --connect-timeout 5 --max-time 30 --retry 3 --retry-all-errors --retry-delay 1 --cacert /tls/tls.crt -XPOST -d "$KEY" https://multi-platform-otp-server.multi-platform-controller.svc.cluster.local/store-key); then


[low] edge-case

The --retry-all-errors flag causes curl to retry on ALL failures, including non-transient HTTP errors like 400 Bad Request or 403 Forbidden. If the OTP server returns a 4xx error, the script will retry 3 times with 1-second delays before failing, adding unnecessary delay.

fullsend-ai-review · 2026-06-22T12:15:07Z

    echo "{message: \"Creating secret file using TLS certificate...\", level: \"INFO\"}"
    KEY=$(cat id_rsa)
-    if ! otp_raw=$(curl --fail --cacert /tls/tls.crt -XPOST -d "$KEY" https://multi-platform-otp-server.multi-platform-controller.svc.cluster.local/store-key); then
+    if ! otp_raw=$(curl --fail --connect-timeout 5 --max-time 30 --retry 3 --retry-all-errors --retry-delay 1 --cacert /tls/tls.crt -XPOST -d "$KEY" https://multi-platform-otp-server.multi-platform-controller.svc.cluster.local/store-key); then


[low] parameter-inconsistency

The curl timeout parameters (--connect-timeout 5 --max-time 30) differ from the existing curl pattern at provision-shared-host.sh:46 (--connect-timeout 10 --max-time 120). The difference is reasonable (internal cluster service vs external GitHub download) but undocumented.

fullsend-ai-review · 2026-06-22T12:15:07Z

    echo "{message: \"Creating secret file using TLS certificate...\", level: \"INFO\"}"
    KEY=$(cat id_rsa)
-    if ! otp_raw=$(curl --fail --cacert /tls/tls.crt -XPOST -d "$KEY" https://multi-platform-otp-server.multi-platform-controller.svc.cluster.local/store-key); then
+    if ! otp_raw=$(curl --fail --connect-timeout 5 --max-time 30 --retry 3 --retry-all-errors --retry-delay 1 --cacert /tls/tls.crt -XPOST -d "$KEY" https://multi-platform-otp-server.multi-platform-controller.svc.cluster.local/store-key); then


[info] consistency

The retry parameters (--retry 3 --retry-all-errors --retry-delay 1) are consistently applied across all three modified files.

glevi-rh · 2026-06-26T20:45:43Z

/fs-review

fullsend-ai-review · 2026-06-26T20:49:27Z

🤖 Finished Review · ✅ Success · Started 8:49 PM UTC · Completed 8:59 PM UTC
Commit: ec21706 · View workflow run →

fullsend-ai-review

See the review comment for full details.

fullsend-ai-review · 2026-06-26T21:08:37Z

🤖 Finished Review · ✅ Success · Started 9:08 PM UTC · Completed 9:17 PM UTC
Commit: ec21706 · View workflow run →

fullsend-ai-review · 2026-06-26T21:17:47Z

  echo "{message: \"Creating secret file using TLS certificate...\", level: \"INFO\"}"
  KEY=$(cat id_rsa)
-  if ! otp_raw=$(curl --fail --cacert /tls/tls.crt -XPOST -d "$KEY" https://multi-platform-otp-server.multi-platform-controller.svc.cluster.local/store-key); then
+  if ! otp_raw=$(curl --fail --connect-timeout 5 --max-time 30 --retry 3 --retry-all-errors --retry-delay 1 --cacert /tls/tls.crt -XPOST -d "$KEY" https://multi-platform-otp-server.multi-platform-controller.svc.cluster.local/store-key); then


[low] inconsistent-retry-pattern

The curl retry flags for the OTP server call (--connect-timeout 5 --max-time 30) differ from the existing curl retry pattern at line 46 (--connect-timeout 10 --max-time 120). The difference is justified (in-cluster OTP POST vs. external GitHub download), but a brief inline comment explaining the rationale would aid future maintainers.

Suggested fix: Consider adding a brief comment explaining why the OTP server curl uses shorter timeouts than the GitHub download curl, to aid future maintainers.

glevi-rh · 2026-06-26T21:20:14Z

/retest

glevi-rh

/lgtm

returning PR to inital state to show fullsend unwanted behavior

fullsend-ai-review Bot approved these changes Jun 22, 2026

View reviewed changes

fullsend-ai-review Bot added the ready-for-merge All reviewers approved — ready to merge label Jun 22, 2026

fullsend-ai-review Bot approved these changes Jun 22, 2026

View reviewed changes

fullsend-ai-review Bot added ready-for-merge All reviewers approved — ready to merge and removed ready-for-merge All reviewers approved — ready to merge labels Jun 22, 2026

fullsend-ai-review Bot suggested changes Jun 26, 2026

View reviewed changes

fullsend-ai-review Bot added bug Something isn't working and removed ready-for-merge All reviewers approved — ready to merge labels Jun 26, 2026

glevi-rh force-pushed the agent/945-otp-curl-retry branch from cfdfb3e to b655611 Compare June 26, 2026 21:05

fullsend-ai-review Bot approved these changes Jun 26, 2026

View reviewed changes

fullsend-ai-review Bot added the ready-for-merge All reviewers approved — ready to merge label Jun 26, 2026

glevi-rh previously approved these changes Jun 26, 2026

View reviewed changes

glevi-rh added hold and removed hold labels Jun 28, 2026

glevi-rh self-requested a review June 28, 2026 16:14

glevi-rh mentioned this pull request Jun 28, 2026

KONFLUX-14156 add curl retry and timeouts to OTP server calls #951

Merged

7 tasks

Uh oh!

Conversation

fullsend-ai-coder Bot commented Jun 22, 2026 • edited by glevi-rh Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Post-script verification

Uh oh!

fullsend-ai-review Bot commented Jun 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fullsend-ai-review Bot commented Jun 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review

Findings

Low

Review

Findings

High

Medium

Low

Review

Findings

Low

Info

Uh oh!

fullsend-ai-review Bot Jun 22, 2026

Choose a reason for hiding this comment

Uh oh!

fullsend-ai-review Bot Jun 22, 2026

Choose a reason for hiding this comment

Uh oh!

manish-jangra commented Jun 22, 2026

Uh oh!

fullsend-ai-coder Bot commented Jun 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fullsend-ai-coder Bot commented Jun 22, 2026

🔧 Fix agent — iteration 1 (human-triggered)

Uh oh!

fullsend-ai-review Bot commented Jun 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fullsend-ai-review Bot Jun 22, 2026

Choose a reason for hiding this comment

Uh oh!

fullsend-ai-review Bot Jun 22, 2026

Choose a reason for hiding this comment

Uh oh!

fullsend-ai-review Bot Jun 22, 2026

Choose a reason for hiding this comment

Uh oh!

glevi-rh commented Jun 26, 2026

Uh oh!

fullsend-ai-review Bot commented Jun 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fullsend-ai-review Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fullsend-ai-review Bot commented Jun 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fullsend-ai-review Bot Jun 26, 2026

Choose a reason for hiding this comment

Uh oh!

glevi-rh commented Jun 26, 2026

Uh oh!

glevi-rh left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

fullsend-ai-coder Bot commented Jun 22, 2026 •

edited by glevi-rh

Loading

fullsend-ai-review Bot commented Jun 22, 2026 •

edited

Loading

fullsend-ai-review Bot commented Jun 22, 2026 •

edited

Loading

fullsend-ai-coder Bot commented Jun 22, 2026 •

edited

Loading

fullsend-ai-review Bot commented Jun 22, 2026 •

edited

Loading

fullsend-ai-review Bot commented Jun 26, 2026 •

edited

Loading

fullsend-ai-review Bot commented Jun 26, 2026 •

edited

Loading