Skip to content

[TENT] Honor forced TCP in classic compatibility layer - #4027

Open
ToLiveAndLove wants to merge 1 commit into
kvcache-ai:mainfrom
ToLiveAndLove:fix/3995-tent-force-tcp
Open

[TENT] Honor forced TCP in classic compatibility layer#4027
ToLiveAndLove wants to merge 1 commit into
kvcache-ai:mainfrom
ToLiveAndLove:fix/3995-tent-force-tcp

Conversation

@ToLiveAndLove

@ToLiveAndLove ToLiveAndLove commented Sep 11, 2026

Copy link
Copy Markdown

Description

Honor the classic Transfer Engine TCP constraints when existing callers run on
TENT through MC_USE_TENT=1.

Classic callers use either MC_FORCE_TCP=1 or
initialize(..., protocol="tcp") as a hard transport constraint. Both signals
were previously dropped by the compatibility path, allowing TENT to initialize
and select RDMA.

This change:

  • translates MC_FORCE_TCP into TENT configuration;
  • forwards the Python protocol argument through an ABI-safe init overload;
  • disables RDMA initialization when TCP is forced;
  • restricts memory-transfer selection to TCP, even when metadata or a custom
    policy lists RDMA first or lists only RDMA;
  • preserves non-memory transport policies.

Both the configuration and selector changes are necessary. Configuration alone
cannot override a conflicting policy. Selector enforcement alone would still
initialize RDMA devices and consume their resources.

Refs #3995 (P0-force-tcp)

Negative validation

On unmodified origin/main, with RDMA-capable hosts:

  • MC_FORCE_TCP=1 still initialized RDMA and attempted an mlx5 endpoint before
    falling back to TCP.
  • initialize(..., protocol="tcp") with an RDMA-only TENT policy loaded
    IbvLoader and initialized mlx5_0 and mlx5_2; TCP was not installed.

Positive validation

With this change:

  • MC_FORCE_TCP=1 installs TCP without creating an RDMA context.
  • protocol="tcp" has the same behavior without requiring MC_FORCE_TCP.
  • an RDMA-only memory policy is overridden, while file policies are unchanged;
  • legacy transport selection cannot bypass the TCP constraint;
  • a conflicting non-TCP request hint fails instead of bypassing the TCP
    constraint.

Module

  • Transfer Engine (mooncake-transfer-engine)
  • Mooncake Store (mooncake-store)
  • Reshard (mooncake-reshard)
  • Mooncake EP (mooncake-ep)
  • Mooncake PG (mooncake-pg)
  • Integration (mooncake-integration)
  • P2P Store (mooncake-p2p-store)
  • Python Wheel (mooncake-wheel)
  • Common (mooncake-common)
  • Mooncake RL (mooncake-rl)
  • CI/CD
  • Docs
  • Other

Type of Change

  • Bug fix
  • New feature
  • Refactor
  • Breaking change
  • Documentation update
  • Performance improvement
  • Other

How Has This Been Tested?

Test commands:

cmake --build build \
  --target tebench engine transport_uint_test \
  transfer_engine_config_override_test tent_transport_selector_test \
  config_lifecycle_test tent_transport_hint_test \
  tent_tcp_datapath_roundtrip_test -j32

ctest --test-dir build \
  -R 'transport_uint_test|config_lifecycle_test|transfer_engine_config_override_test|tent_transport_selector_test|tent_transport_hint_test|tent_tcp_datapath_roundtrip_test' \
  --output-on-failure

Test results:

  • Unit tests pass
  • Integration tests pass (if applicable)
  • Manual testing done (describe below)

Manual validation used two RDMA-capable hosts:

  • MC_FORCE_TCP=1: repeated WRITE+READ operations passed data verification.
  • protocol="tcp" without MC_FORCE_TCP, with an RDMA-only configured policy:
    repeated 4 MiB WRITE+READ operations passed data verification.
  • ss confirmed established TCP connections on both peers.
  • Neither peer logged RDMA context initialization, endpoint attempts, or
    fallback.

A separate USE_TENT=OFF Release build of transfer_engine also passes.

Checklist

  • I have performed a self-review of my own code
  • I have formatted my code using ./scripts/code_format.sh
  • I have run pre-commit on the files changed in this PR and all hooks pass
  • I have updated the documentation (if applicable)
  • I have added tests to prove my changes are effective
  • For changes >500 LOC: I have filed an RFC issue

AI Assistance Disclosure

  • No AI tools were used
  • AI tools were used (specify below)

AI assistance was used. The author reviewed the code and ran the reported
tests.

Comment on lines +144 to +160
TEST(TransferEngineTentCompatibilityTest, TcpProtocolForcesTcpTransport) {
ScopedEnvVar use_tent("MC_USE_TENT", "1");
ScopedEnvVar force_tcp("MC_FORCE_TCP", nullptr);
ScopedEnvVar hostname("MOONCAKE_LOCAL_HOSTNAME", "127.0.0.1");
ScopedEnvVar conf(
"MC_TENT_CONF",
R"({"transports":{"tcp":{"enable":false},"rdma":{"enable":false},"shm":{"enable":false},"hp_tcp":{"enable":false},"mpcomm":{"enable":false},"io_uring":{"enable":false}},"metrics":{"enabled":false}})");

TransferEngine engine(true);
ASSERT_TRUE(engine.isUsingTent());
ASSERT_EQ(engine.init(P2PHANDSHAKE, "compat-protocol-tcp", "", 0, "tcp"),
0);

std::array<char, 4096> buffer{};
ASSERT_EQ(engine.registerLocalMemory(buffer.data(), buffer.size()), 0);
EXPECT_EQ(engine.unregisterLocalMemory(buffer.data()), 0);
}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TcpProtocolForcesTcpTransport currently only checks that init(..., "tcp") returns 0 and that register/unregister succeed. That would still pass if protocol were ignored again, as long as the engine comes up with some transport.

This test is the only coverage of the classic shim path (TransferEngine::init + MC_USE_TENT=1). Could we make the assertion match the bug?

  • Also cover MC_FORCE_TCP=1 with protocol="" (that shim path has no unit test today; LegacyForceTcpEnvOverridesTentConfig only hits ConfigHelper::loadFromEnv).
  • After init, assert the TCP constraint actually stuck — e.g. transports/rdma/enable == false / transports/force_tcp == true, or that RDMA was not installed. Register/unregister alone does not prove we stopped selecting RDMA.

Without that, a regression that drops forceTcp() in the 5-arg init overload would still look green.

@ToLiveAndLove

Copy link
Copy Markdown
Author

The only failing check, build-wheel-rocm (3.10), was terminated by the runner with exit code 137 and The runner has received a shutdown signal at build step 234/258. There is no compiler error in the log; the ROCm Python 3.12 job and all TENT checks passed. Could a maintainer rerun the failed job?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants