feat(api): add immutability for TrainingRuntimes types

[Misha6Sharma]

What this PR does / why we need it:
This PR implements immutability for TrainingRuntime and ClusterTrainingTime to prevent any modifications after its creation
Used CRD Markers and Webhooks for the same for the .spec and .metadata(labels and annotations) fields

Which issue(s) this PR fixes (optional, in Fixes #<issue number>, #<issue number>, ... format, will close the issue(s) when PR gets merged):
Fixes #2599

Checklist:

• Docs included if any changes are user facing

[github-actions]

🎉 Welcome to the Kubeflow Trainer! 🎉

Thanks for opening your first PR! We're happy to have you as part of our community 🚀

Here's what happens next:

• If you haven't already, please check out our Contributing Guide for repo-specific guidelines and the Kubeflow Contributor Guide for general community standards.
• Our team will review your PR soon! cc @kubeflow/kubeflow-trainer-team

Join the community:

• Slack: Join our #kubeflow-trainer Slack channel.
• Meetings: Attend the Kubeflow AutoML and Training Working Group bi-weekly meetings.

Feel free to ask questions in the comments if you need any help or clarification!
Thanks again for contributing to Kubeflow! 🙏

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign gaocegege for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[andreyvelich]

Thanks @Misha6Sharma, we are going to discuss this as part of this KEP: #3428
/close
cc @robert-bell

[google-oss-prow]

@andreyvelich: Closed this PR.

Details

In response to this:

Thanks @Misha6Sharma, we are going to discuss this as part of this KEP: #3428
/close
cc @robert-bell

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.