Skip to content

fix(python): resolve real interpreter for Process runner without pythonVersion#394

Merged
jymaire merged 1 commit into
mainfrom
fix/python-process-runner-interpreter
Jul 8, 2026
Merged

fix(python): resolve real interpreter for Process runner without pythonVersion#394
jymaire merged 1 commit into
mainfrom
fix/python-process-runner-interpreter

Conversation

@jymaire

@jymaire jymaire commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

What changes are being made and why?

Python Script/Commands tasks running on the Process task runner failed with /bin/sh: python: not found (exit code 127) on hosts that ship only python3 (Debian/Ubuntu, and the standard Kestra worker image).

Root cause: In PythonEnvironmentManager.setup(), the interpreter was only resolved to an actual installed binary when pythonVersion was explicitly set. When it was unset (the common case) on a Process runner, the interpreter stayed the literal "python", which doesn't exist on python3-only hosts. The Docker default runner was unaffected because python:3.13-slim symlinks python.

Fix: The Process runner now always resolves an installed interpreter. When pythonVersion is set, behavior is unchanged (resolved through the configured package manager, which may provision a managed Python via uv). When it's unset, we probe for an already-installed interpreter (python3python) via PackageManagerType.PIP.getPythonPath, avoiding a managed-Python download just to run a dependency-free script. The Docker/Kubernetes path is untouched.

Also:

  • Updated the all_python sanity-check flow: python_commands_process now uses python3 main.py (user-authored command, not routed through the resolver).
  • Added PythonEnvironmentManagerTest regression test asserting a Process-runner Script with no pythonVersion resolves to a runnable, non-"python" interpreter.

How the changes have been QAed?

  • New PythonEnvironmentManagerTest passes.
  • ScriptTest.task[PROCESS] — the exact case that reproduced the reported failure — now passes (previously exit 127).
id: repro
namespace: sanitychecks.plugin-scripts
tasks:
  - id: python_script_process
    type: io.kestra.plugin.scripts.python.Script
    taskRunner:
      type: io.kestra.plugin.core.runner.Process
    script: |
      print("Hello from a Process Task Runner")

On a python3-only host this failed with python: not found before the fix and now succeeds.


Contributor Checklist ✅

…onVersion

Python Script/Commands tasks using the Process task runner failed with
"/bin/sh: python: not found" (exit 127) on hosts that ship only python3
(Debian/Ubuntu, the standard Kestra worker image). The interpreter was only
resolved to an installed binary when pythonVersion was explicitly set;
otherwise it stayed the literal "python".

Now the Process runner always resolves an installed interpreter, probing
python3 before python via PackageManagerType.PIP.getPythonPath when no
pythonVersion is set, without triggering a managed-Python download for the
no-dependency case. The Docker/K8s path is unchanged.

Also updates the all_python sanity-check flow (python3 main.py) and adds a
regression test.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@jymaire jymaire self-assigned this Jul 7, 2026
@github-actions

github-actions Bot commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

📦 Artifacts

Name Size Updated Expiration
jar 56.71 MB Jul 7, 26, 2:37:39 PM UTC Jul 14, 26, 2:37:36 PM UTC

🧪 Java Unit Tests

TestsPassed ✅SkippedFailedTime ⏱
Java Tests Report239 ran239 ✅0 ⚠️0 ❌24m 44s 888ms

🔁 Unreleased Commits

✅ No unreleased commits found.

@github-actions

github-actions Bot commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Tests report quick summary:

success ✅ > tests: 239, success: 239, skipped: 0, failed: 0

unfold for details
Project Status Success Skipped Failed
plugin-script-bun success ✅ 3 0 0
plugin-script-deno success ✅ 3 0 0
plugin-script-dotnet success ✅ 8 0 0
plugin-script-go success ✅ 18 0 0
plugin-script-groovy success ✅ 8 0 0
plugin-script-jbang success ✅ 2 0 0
plugin-script-julia success ✅ 2 0 0
plugin-script-jython success ✅ 5 0 0
plugin-script-lua success ✅ 3 0 0
plugin-script-nashorn success ✅ 6 0 0
plugin-script-node success ✅ 13 0 0
plugin-script-perl success ✅ 3 0 0
plugin-script-php success ✅ 3 0 0
plugin-script-powershell success ✅ 4 0 0
plugin-script-python success ✅ 70 0 0
plugin-script-r success ✅ 2 0 0
plugin-script-ruby success ✅ 39 0 0
plugin-script-shell success ✅ 47 0 0

@jymaire

jymaire commented Jul 7, 2026

Copy link
Copy Markdown
Contributor Author

QA

Flow

id: test_python_process_fix
namespace: sanitychecks.plugin_scripts

tasks:
    # ── THE FIX ──────────────────────────────────────────────────────────
    # Script on the Process runner, no pythonVersion.
    # BEFORE the fix: "/bin/sh: python: not found" (exit 127) on python3-only hosts.
    # AFTER: resolves python3 and prints its path.
    - id: script_process_default
      type: io.kestra.plugin.scripts.python.Script
      taskRunner:
        type: io.kestra.plugin.core.runner.Process
      script: |
        import sys
        print(f"Resolved interpreter: {sys.executable}")
        print(f"Version: {sys.version.split()[0]}")
        print("OK: ran on Process runner with no pythonVersion")

    # Same path, forcing PIP — this is the exact method the fix calls
    # (PackageManagerType.PIP.getPythonPath → probes python3 then python).
    - id: script_process_pip
      type: io.kestra.plugin.scripts.python.Script
      taskRunner:
        type: io.kestra.plugin.core.runner.Process
      packageManager: PIP
      script: |
        import sys
        print(f"Resolved interpreter: {sys.executable}")

    # Explicit pythonVersion on Process runner — unchanged behavior, must still work.
    - id: script_process_explicit_version
      type: io.kestra.plugin.scripts.python.Script
      taskRunner:
        type: io.kestra.plugin.core.runner.Process
      pythonVersion: "3.12"
      script: |
        import sys
        print(f"Resolved interpreter: {sys.executable}")

    # ── CONTROL ──────────────────────────────────────────────────────────
    # Docker default runner (python:3.13-slim) — always worked; must not regress.
    - id: script_docker_default
      type: io.kestra.plugin.scripts.python.Script
      script: |
        print("OK: ran on Docker runner")

Screen

image

@jymaire

jymaire commented Jul 8, 2026

Copy link
Copy Markdown
Contributor Author

🧪 QA Report — ✅ PASS

Edition: OSS | Branch: develop (2.0.0-SNAPSHOT)

Verified end-to-end in the Kestra UI (Process runner executes on the worker host). This QA host has python3 but no python (which python → not found), i.e. it reproduces the exact "python3-only" condition from the bug report. Both tasks that were red in the original report now pass.

Flow Result
test_python_process_fix (focused) ✅ SUCCESS — 3/3 tasks
all_python (repo sanity flow) ✅ SUCCESS — 9/9 tasks incl. assert

Before → after (root cause): previously the plugin built the command as python /…/script.py, which on a python3-only host fails with /bin/sh: python: not found (exit 127). After the fix, the Process runner resolves a real interpreter — the executed command is now python3.14 /…/script.py.


Flow 1: test_python_process_fix (✅ SUCCESS)

Flow YAML
id: test_python_process_fix
namespace: sanitychecks.plugin_scripts
tasks:
  - id: script_process_default        # Script + Process, no pythonVersion — THE fix
    type: io.kestra.plugin.scripts.python.Script
    taskRunner:
      type: io.kestra.plugin.core.runner.Process
    script: |
      import sys
      print(f"Resolved interpreter: {sys.executable}")
      print(f"Version: {sys.version.split()[0]}")
      print("OK: ran on Process runner with no pythonVersion")
  - id: script_process_pip             # Process + packageManager PIP (exact method the fix calls)
    type: io.kestra.plugin.scripts.python.Script
    taskRunner:
      type: io.kestra.plugin.core.runner.Process
    packageManager: PIP
    script: |
      import sys
      print(f"Resolved interpreter (PIP): {sys.executable}")
  - id: script_docker_default          # Docker runner — control / non-regression
    type: io.kestra.plugin.scripts.python.Script
    script: |
      print("OK: ran on Docker runner (control / non-regression)")

Gantt

Task Runner Status Duration
script_process_default Process (no pythonVersion) SUCCESS 11.04s
script_process_pip Process (PIP) SUCCESS 0.07s
script_docker_default Docker SUCCESS 0.72s
Total SUCCESS 12.01s

Logs synthesis

  • script_process_default → executed python3.14 /…/script.py; logged Resolved interpreter: /usr/bin/python3.14, Version: 3.14.4. No python: not found.
  • script_process_pip → executed python3.14 …; Resolved interpreter (PIP): /usr/bin/python3.14.
  • script_docker_default → executed python /…/script.py inside the container (unchanged Docker path — image symlinks python), confirming no regression.

Flow 2: all_python (✅ SUCCESS) — literal reproduction of the reported failure

Flow YAML (repo sanity-check flow, with the python3 main.py fix)
# plugin-script-python/src/test/resources/sanity-checks/all_python.yaml
# Key tasks (the two that failed in the original report):
  - id: python_script_process
    type: io.kestra.plugin.scripts.python.Script
    taskRunner:
      type: io.kestra.plugin.core.runner.Process
    script: |
      print("Hello, World from a Process Task Runner")
  - id: python_commands_process
    type: io.kestra.plugin.scripts.python.Commands
    taskRunner:
      type: io.kestra.plugin.core.runner.Process
    inputFiles:
      main.py: |
        print("Hello, World from the Commands Task and Process Task Runner")
    commands:
      - python3 main.py     # <-- changed from `python main.py`

Gantt

Task Status Duration
python_script SUCCESS 0.61s
python_script_process SUCCESS 0.10s
python_commands SUCCESS 0.49s
python_commands_process SUCCESS 0.09s
python_script_logs SUCCESS 25.40s
python_script_metrics SUCCESS 21.44s
python_script_outputs SUCCESS 13.85s
python_commands_etl_outputs SUCCESS 1.34s
assert SUCCESS 0.04s
Total SUCCESS ~65s

⭐ = tasks that failed with python: not found (exit 127) in the original report.

Logs synthesis

  • python_script_process → executed python3.14 /…/script.pyHello, World from a Process Task Runner.
  • python_commands_process → executed python3 main.pyHello, World from the Commands Task and Process Task Runner.
  • assert passed all exitCode == 0 conditions; python_commands_etl_outputs produced processed_orders.csv.

Outputs synthesispython_script_outputs and python_commands_etl_outputs returned the total output and the processed_orders.csv output file, as asserted.


Timeouts / notes: none. The _logs/_metrics/_outputs tasks take 13–25s (network: pip install, HuggingFace dataset download, pydata image) but all completed and are unrelated to this fix. Topology/artifact testing not applicable (this PR does not add plugin artifacts).

@jymaire jymaire merged commit 28e6c0d into main Jul 8, 2026
9 checks passed
@jymaire jymaire deleted the fix/python-process-runner-interpreter branch July 8, 2026 06:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants