Arm backend: Add manifest validation and add Backwards compabillety testing

[Sebastian-Larsson]

cc @digantdesai @freddan80 @per @zingo @oscarandersson8218 @mansnils @robell @rascani

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/18813

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

• [ROCm] MI350 CI jobs will have longer queue times due to CI migration

❌ 8 New Failures, 2 Pending, 2 Unrelated Failures

As of commit 9bf4cad with merge base ef5c8a7 ():

NEW FAILURES - The following jobs have failed:

• Cadence Build & Test / hifi-build / hifi4 (gh)
  Input required and not supplied: aws-region
• Cadence Build & Test / vision-build / vision (gh)
  Input required and not supplied: aws-region
• pull / android / run-emulator (gh)
  The process '/usr/local/lib/android/sdk/platform-tools/adb' failed with exit code 224
• pull / test-arm-backend-no-driver (test_pytest_models_tosa) / linux-job (gh)
  RuntimeError: Command docker exec -t f0f1bfb13b31026c47996a610ecc826881653e3b3b44126660de5925e3b21be2 /exec failed with exit code 1
• pull / test-llama-runner-qnn-linux (fp32, qnn_16a16w, qnn) / linux-job (gh)
  RuntimeError: Command docker exec -t 1f3df29c2881caeadb106a3968a2cf7e00e24bd197dbe5c5bdefac4f81224c43 /exec failed with exit code 92
• trunk / test-arm-backend-ethos-u (test_pytest_models_ethos_u55) / linux-job (gh)
  RuntimeError: Command docker exec -t bf8456d4066c969998636ed3aeb39946397c57c71d1af1ef528bff19b42e195a /exec failed with exit code 1
• trunk / test-arm-backend-ethos-u (test_pytest_models_ethos_u85) / linux-job (gh)
  RuntimeError: Command docker exec -t 1f023aeb9ba49d1cb12567291c849cd67375ad289e5cb3e667dbf772ebaa1f50 /exec failed with exit code 1
• trunk / test-arm-backend-vkml (test_pytest_models_vkml) / linux-job (gh)
  RuntimeError: Command docker exec -t 43e3e48cc6cd6352154e3f835d671e516ec3664e005c209b0c42a6668d00104a /exec failed with exit code 1

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

• pull / test-qnn-testsuite-linux / test-backend-linux (qnn, models) / linux-job (gh) (detected as infra flaky with no log or failing log classifier)
• trunk / test-llama-runner-qnn-linux (fp32, qnn_16a16w, qnn) / linux-job (gh) (detected as infra flaky with no log or failing log classifier)

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[zingo]

Hi @digantdesai we tried to handle the buck2 files with this, but you might want to take a peek anyway.

[zingo]

@digantdesai this adds some files and update buck2 stuff so you might want to check this :)

[GregoryComer]

Thanks. Buck CI is running internally now and I'll report back if there's anything issues.

[meta-codesync]

@GregoryComer has imported this pull request. If you are a Meta employee, you can view this in D101685482.

[digantdesai]

I know this is not the PR introducing this API checker, but two questions,
(1) why not rely on unit-tests to enforce the actual behavior, beyond API BC
(2) why not use Griffe or something similar to do this for you?

[Sebastian-Larsson]

1. Adding the validation scripts lints the API surface for mismatches, leading to early feedback when something isn't adhering to the API. We can also control deprecation and experimental features easily. Also, with a testing only approach we'll miss when the API is updated and no tests are added for it. That would require a check of its own.
2. Adding these scripts was relatively simple and let's us control the entire flow. Griffe may be able to replace parts of this, but not all, meaning we would still need custom code to handle our API.

[digantdesai]

Ok with this PR, but left a couple of question for the original motivation (sorry I was out for a few months)

[GregoryComer]

The internal buck build is complaining about the following: error: fail: src files must be named test_.py, _test.py, or conftest.py: public_api_bc/ethosu_flow.py. Seems like an odd constraint, but would it be possible to rename this file? CC @digantdesai

[zingo]

OK to merge :)

[digantdesai]

Rebase please, I think the test file rename should work but I can pull it internal once more before merging to avoid a silly-issue revert. Thanks.

[digantdesai]

There were some tooling issues, imported now

[digantdesai]

perhaps needs some buck fixes
"""
executorch/backends/arm/test/public_api_bc/test_ethosu_flow.py:11: in
from executorch.backends.arm import (
E ImportError: cannot import name 'EthosUBackend' from 'executorch.backends.arm'
"""

[Sebastian-Larsson]

perhaps needs some buck fixes """ executorch/backends/arm/test/public_api_bc/test_ethosu_flow.py:11: in from executorch.backends.arm import ( E ImportError: cannot import name 'EthosUBackend' from 'executorch.backends.arm' """

@digantdesai I've done a best effort attempt to implement the BUCK2 fixes. Please import to check if they are sufficient. Thanks!

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: Sebastian-Larsson / name: Sebastian Larsson (e541616, 67dc7b9, 7e60f85, facecbb)

[digantdesai]

Can you please rebase?

[digantdesai]

From internal CI

executorch/backends/arm/test/public_api_bc/test_vgf_fp_flow.py:13: in <module>
    from executorch.extension.export_util.utils import save_pte_program
E   ModuleNotFoundError: No module named 'executorch.extension.export_util'

[Sebastian-Larsson]

I keep getting merge conflicts and having to rebase. Let me know if you need me to rebase again so that you can check internally.