Skip to content

Hugov/mmap taskhandler#652

Open
HugoVuach wants to merge 4 commits into
aneoconsulting:mainfrom
HugoVuach:hugov/mmap-taskhandler
Open

Hugov/mmap taskhandler#652
HugoVuach wants to merge 4 commits into
aneoconsulting:mainfrom
HugoVuach:hugov/mmap-taskhandler

Conversation

@HugoVuach
Copy link
Copy Markdown

Motivation

The current TaskHandler eagerly loads the payload and all data dependencies into memory, which can be costly for large inputs. This PR refactors the implementation to reduce copies and memory usage while keeping backward compatibility.

Description

  • Introduces lazy loading of payload and data dependencies using mmap.

  • Adds zero-copy APIs:

    • getPayloadView() returning an absl::string_view
    • getDataDependenciesView() returning std::map<std::string, absl::string_view>
  • Keeps existing APIs (getPayload(), getDataDependencies()) by building owning caches on demand.

  • Refactors result writing: replaces ofstream with mmap write + memcpy + msync, keeping mappings alive until TaskHandler destruction.

  • Adds centralized cleanup of mappings (RAII via destructor).

Testing

  • Built the C++ worker API after changes.
  • Manual checks: payload/dependencies content matches between *View() and owning getters; result file is written and NotifyResultData succeeds.

Impact

  • Lower memory usage and fewer copies (especially for large payloads/dependencies).
  • New POSIX mmap usage (portability consideration for non-POSIX platforms).
  • Behavior is backward compatible; absl::string_view lifetimes are tied to the TaskHandler instance.

Additional Information

Checklist

  • My code adheres to the coding and style guidelines of the project.
  • I have performed a self-review of my code.
  • I have commented my code, particularly in hard-to-understand areas.
  • I have made corresponding changes to the documentation.
  • I have thoroughly tested my modifications and added tests when necessary.
  • Tests pass locally and in the CI.
  • I have assessed the performance impact of my modifications.

@CLAassistant
Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.


Hugo Vuacheux seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
You have signed the CLA already but the status is still pending? Let us recheck it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants