Add Core ML video tracking export

[AhmadDurrani579]

Summary

• add an iOS 18 Core ML export for EdgeTAM's temporal video-tracking pipeline
• export separate image encoder, prompt initializer, memory encoder, and prompt-free propagator models
• maintain conditioning memory, recent memories, and object pointers in an explicit client-owned memory bank
• provide a Python predictor, PyTorch/Core ML parity validator, documentation, and focused tests

Motivation

The existing Core ML path supports prompted image segmentation, but it does not export EdgeTAM's temporal memory pipeline for real video tracking. Re-running image segmentation on each frame does not preserve the video predictor's spatial memories or object-pointer history.

This change keeps the exported models stateless and moves the small fixed-shape memory bank into the client. That avoids a dependency on stateful Core ML operations while preserving prompt-free propagation after the first frame.

Exported models

• EdgeTAMVideoImageEncoder.mlpackage
• EdgeTAMVideoInitializer.mlpackage
• EdgeTAMVideoMemoryEncoder.mlpackage
• EdgeTAMVideoPropagator.mlpackage

Generated model packages are intentionally excluded from the repository.

Validation

PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 \
PYTHONDONTWRITEBYTECODE=1 \
PYTHONPATH=coreml/video_tracking \
python -B -m pytest coreml/video_tracking/tests -q

Result: 33 passed.

The included validator also compares per-frame Core ML output with the official PyTorch video predictor using mask IoU, logit cosine similarity, mean absolute error, and maximum absolute error.

Current scope

• single-object, forward-only tracking
• one to four first-frame prompt tokens
• fixed 1024-by-1024 model input
• one conditioning memory, six recent memories, and sixteen object pointers
• no mid-video correction, reverse propagation, quantization, or bundled Swift wrapper

[meta-cla]

Hi @AhmadDurrani579!

Thank you for your pull request and welcome to our community.

Action Required

In order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at cla@meta.com. Thanks!

[meta-cla]

Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Meta Open Source project. Thanks!

[AhmadDurrani579]

Hi maintainers, I wanted to gently follow up on this contribution when you have availability.
The branch remains synchronized with main, the Meta CLA is signed, and the focused test suite reports 33 passing tests. Generated Core ML packages are excluded from the repository.
I would appreciate feedback on whether this stateless four-model architecture and explicit memory-bank approach fit the project’s direction. I’m happy to address feedback, reduce the scope, or split the contribution if that would make review easier.
Thank you for considering it