Bundle tokenizer entrypoint as ExecuTorch delegate

[JacobSzwejbka]

Summary:

• add a manual ExecuTorch tokenize execution plan backed by TokenizerBackend
• bundle tokenizer bytes as inline delegate data and wire LLM export to append the entrypoint
• add runtime buffer loading for tokenizers and string input rebinding

Submodule:

• updates extension/llm/tokenizers to meta-pytorch/tokenizers@def7094521 on branch js/tokenizer-delegate-entrypoint

Test Plan:

• cmake --build cmake-out --target extension_llm_runner -j8
• python -m py_compile extension/llm/export/tokenizer_delegate.py extension/llm/export/test/test_tokenizer_delegate.py
• python -m pytest extension/llm/export/test/test_tokenizer_delegate.py -q could not collect in this environment because local torchao imports expect torch.ops.higher_order.templated_attention

Notes:

• commit hook was bypassed because lintrunner_adapters is missing locally (python: No module named lintrunner_adapters).

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/20173

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

This comment was automatically generated by Dr. CI and updates every 15 minutes.