fix(BA-4624): Validate cloud detection responses and harden metadata JSON parsing

[seedspirit]

resolves BA-4624

Checklist: (if applicable)

• Milestone metadata specifying the target backport version
• Mention to the original issue
• Installer updates including:
  • Fixtures for db schema changes
  • New mandatory config options
• Update of end-to-end CLI integration tests in ai.backend.test
• API server-client counterparts (e.g., manager API -> client SDK)
• Test case(s) to:
  • Demonstrate the difference of before/after
  • Demonstrate the flow of abstract/conceptual models with a concrete implementation
• Documentation
  • Contents in the docs directory
  • docstrings in public interfaces and type annotations

[Copilot]

Pull request overview

This PR hardens cloud-provider detection and instance metadata parsing to avoid false positives when running on non-major cloud providers, and adds unit tests to cover the new validation behavior.

Changes:

• Validate AWS/Azure/GCP IMDS responses in _detect_* functions and tighten detect_cloud() timeouts/logging.
• Make instance metadata JSON parsing resilient to empty/invalid payloads across AWS/Azure/GCP helpers.
• Add unit tests for IMDS detection and JSON parsing fallbacks; add a towncrier fix fragment.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File	Description
tests/unit/common/test_identity.py	Adds new unit tests for cloud detection validation and metadata parsing fallback paths.
src/ai/backend/common/identity.py	Implements IMDS response validation, tighter client timeouts, and safer JSON parsing for instance metadata helpers.
changes/9653.fix.md	Adds a release note fragment describing the fix.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

[jopemachine]

This change is not part of this PR, but it would be better to extract 169.254.169.254 into a variable instead of repeating it.

[HyeockJinKim]

Please check the CI test. @seedspirit