Skip to content

Give partitioner an arg for is_executable() #84

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
noah-weingarden merged 3 commits into from
Apr 28, 2025
Merged

Give partitioner an arg for is_executable() #84

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
8f54e7d
Give partitioner an arg for is_executable()
noah-weingarden Apr 18, 2025
ababb64
Add comment, pass actual num_reducers
noah-weingarden Apr 24, 2025
da3a0ac
Empty commit because CI didn't run for some reason
noah-weingarden Apr 28, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions madoop/mapreduce.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,8 +43,9 @@ def mapreduce(
# Executable scripts must have valid shebangs
is_executable(map_exe)
is_executable(reduce_exe)
# The partitioner executable expects to receive num_reducers as an arg
if partitioner:
is_executable(partitioner)
is_executable(partitioner, str(num_reducers))

# Create a tmp directory which will be automatically cleaned up
with tempfile.TemporaryDirectory(prefix="madoop-") as tmpdir:
Expand Down Expand Up @@ -157,7 +158,7 @@ def normalize_input_paths(input_path):
return input_paths


def is_executable(exe):
def is_executable(exe, *args):
"""Verify exe is executable and raise exception if it is not.

Execute exe with an empty string input and verify that it returns zero. We
Expand All @@ -168,7 +169,7 @@ def is_executable(exe):
exe = pathlib.Path(exe).resolve()
try:
subprocess.run(
str(exe),
[str(exe), *args],
shell=False,
input="".encode(),
stdout=subprocess.PIPE,
Expand Down