Add support for ~/.databrickscfg

[martin-walsh]

To aid deployment in CI environments it would be helpful to support .databrickscfg as described here https://docs.databricks.com/aws/en/dev-tools/auth/config-profiles

[anayn-sm]

To add context to what @martin-walsh mentioned, In addition to the .databrickscfg file, pytester currently requires that the DATABRICKS_HOST env variable should be explicitly set.

pytester/src/databricks/labs/pytester/fixtures/baseline.py

Line 152 in 64fb32f

host=debug_env["DATABRICKS_HOST"],

If I have a default profile in my .databrickscfg file, the DATABRICKS_HOST env variable shouldn't be required.

The example specified on https://databricks-sdk-py.readthedocs.io/en/latest/authentication.html page works without specifying the host explicitly
w = WorkspaceClient()

Suggested fix:

 return WorkspaceClient(
       host=debug_env.get("DATABRICKS_HOST"), # use .get instead of direct indexing
       auth_type=debug_env.get("DATABRICKS_AUTH_TYPE"),
       ...
      )

[NeyasG]

Chiming in to support this feature request. Currently using the pytester plugin for our integration tests, however it's the only tool in our chain which doesn't support the .databrickscfg file, making authentication and the local developer experience a bit confusing at times.

If there is a way to use the native databricks-cli authentication, which uses the .databrickscfg profiles, much like the databricks-connect + pytester functionality specified here https://docs.databricks.com/aws/en/dev-tools/vscode-ext/pytest#run-tests-using-databricks-connect that would be amazing.