Azure Bastion Integration with AKS - Az AKS CLI integration

[mayasingh17]

Azure Bastion Integration with AKS enables customers to access private clusters using Az AKS Bastion via a Bastion Tunnel. AKS customers who deploy private clusters or public clusters with API server authorized IP ranges often have a need to access their cluster API server from outside the usual scope – whether this is developers who are working from home, a private cluster deployed in an island virtual network. Traditional solutions like VPN servers and jump boxes can be cumbersome and costly to set up and maintain. With Azure Bastion support for AKS, Azure Bastion can now be used to provide a secure tunnel from the local machine to the AKS API server, allowing users to utilize their regular Kubernetes tooling seamlessly. This integration simplifies access, reduces setup time, and enhances security by eliminating the need for exposing public endpoints

[mayasingh17]

https://learn.microsoft.com/en-us/azure/bastion/bastion-connect-to-aks-private-cluster, we are working to enable clusters with public FQDN disabled

[mayasingh17]

We are eager to hear your feedback on this feature - please let us know how this is working for you in the comments of this issue.

[audunsolemdal]

We are eager to hear your feedback on this feature - please let us know how this is working for you in the comments of this issue.

The feature seems very promising apart from a few things:

1. support clusters with public FQDN disabled, which you are working on (great!)
2. Would also like this bug to be fixed, else I will still prefer using a cheap jumpbox VM where my shell has aliases set up.
3. The azure cli commands to run when the bastion host and the AKS cluster are in two different subscriptions, I was able to get this to work, but it was quite unintuitive.

[stevekuznetsov]

@mayasingh17 small feedback - the learn.microsoft.com docs should be updated with how to set up the bastion host for AKS, not just assuming everyone has done it - most of the previous azure bastion examples also use a VM jumpbox which is obviously not necessary now, so a clear .bicep/Infrastructure-as-Code example would be great. I implemented such in this PR in case you want some example. It would also be super useful if we could specify the exact Azure RBAC permissions required to use this feature - the user needs to have AKS permissions to get credentials, and some others to establish the bastion, but it is not clear to me which.

[mbifeld]

Thanks for the feedback @stevekuznetsov. Will work on updating the docs with your suggestions. Also, the PowerShell bug is fixed, and Bastion now works with clusters with public FQDN disabled.

[audunsolemdal]

and Bastion now works with clusters with public FQDN disabled.

Are there any caveats to get this working? I've upgraded my azure CLI + extensions to the latest versions.

Bastion towards my private cluster with public FQDN enabled, my setup works without issues. For my private cluster without public FQDN enabled I end up with this:

k get pods
E1211 07:58:03.488365   35064 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://localhost:52560/api?timeout=32s\": net/http: TLS handshake timeout"

[mbifeld]

Hi @audunsolemdal, the only requirement is having your Bastion be up to date, which is done automatically by Microsoft.

If you're comfortable, can you please send your AKS cluster and Bastion resource IDs to me at mbifeld@microsoft.com so we can investigate?

[audunsolemdal]

If you're comfortable, can you please send your AKS cluster and Bastion resource IDs to me at mbifeld@microsoft.com so we can investigate?

For the public - the error was cryptic, but the reasoning was that Bastion in my network setup didn't resolve the private DNS.
Creating a direct vnet link between the bastion VNET and the privatelink..azmk8s.io seems to do the trick.

The issue was more clear in the Azure Portal via the Bastion --> Connection troubleshoot view, where connecting via the private IP works, but connecting via the private FQDN fails to resolve.