Skip to content

[FIX] Forward the extraction thread count to the integration-test notebook - #558

Draft
noel-improv wants to merge 1 commit into
awslabs:mainfrom
noel-improv:fix/forward-extraction-thread-count
Draft

noel-improv wants to merge 1 commit into
awslabs:mainfrom
noel-improv:fix/forward-extraction-thread-count

Conversation

@noel-improv

Copy link
Copy Markdown
Collaborator

Description

build-tests.sh writes the extraction settings a run needs into the notebook's .env.testing. #490 added EXTRACTION_NUM_WORKERS and EXTRACTION_BATCH_SIZE there, but not EXTRACTION_NUM_THREADS_PER_WORKER, so every run extracted at the default of four threads regardless of what was set locally. This forwards it the same way as its siblings and documents it in env.template.

Changes

  • integration-tests/build-tests.sh: write EXTRACTION_NUM_THREADS_PER_WORKER to .env.testing when it is set, using the same printf %q form as the settings next to it.
  • integration-tests/env.template: add EXTRACTION_NUM_THREADS_PER_WORKER with its default of 4.

Problem

A thread or worker sweep cannot vary the setting it is measuring. It showed up on the single-worker drain re-run for #493, which needed 32 threads and would otherwise have run the 4-thread arm.

Related issue (if any): #493

Testing

  • Unit tests added/updated
  • Integration tests added (as appropriate)
  • Existing tests pass (pytest)
  • Tested manually (describe below)

bash -n passes. Running the new block on its own writes export EXTRACTION_NUM_THREADS_PER_WORKER=32 when the variable is set and writes nothing when it is unset. The same line carried a real run: a stack launched with EXTRACTION_NUM_THREADS_PER_WORKER=32, and the notebook's shipped .env.testing contained that value.

Checklist

  • Code follows existing style and conventions
  • License headers present on new files
  • Documentation updated (if applicable)
  • No breaking changes (or clearly documented)

No new files. Unset, the script behaves as before.


By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

…st notebook

build-tests.sh writes EXTRACTION_NUM_WORKERS and EXTRACTION_BATCH_SIZE into the
notebook's environment file, but not EXTRACTION_NUM_THREADS_PER_WORKER, so a
run always extracted at the default of four threads whatever was set locally.
A thread or worker sweep could not vary the setting it was measuring.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant