How to configure dbt-fabricspark to spin up a Jupyter session via spark_config if this is possible?

[lkozdron-laminar]

Hello

I'm trying to configure my profiles.yml to have dbt-fabricspark spin up a Livy session of type "JupyterSession" with specific runtime information (jupyter2.0), instead of the default "SparkSession". I want to know if this possible to run python notebook with dbt-fabricspark (not pyspark session)

My understanding, based on the adapter's code (specifically livysession.py and credentials.py), is that the spark_config dictionary in profiles.yml is directly passed as the JSON payload for Livy session creation.

I've made the following attempts:

1. Using standard Livy kind parameter:

spark_config: name: my-dbt-session kind: jupyter conf: spark.dbt.runtimeVersion: jupyter2.0

This did not result in a Jupyter session.

2. Using Fabric-specific jobType parameter:
   I found documentation (e.g., from microsoft/fabric-samples/docs-samples/data-engineering/Livy-API-swagger/swagger.json, lines 1501-1508) indicating a jobType parameter with enum values like JupyterEnvironment. I then tried configuring my profiles.yml with:

spark_config: name: my-dbt-session jobType: JupyterEnvironment conf: spark.dbt.runtimeVersion: jupyter2.0

However, the session that spins up is still consistently a "SparkSession" and not a "JupyterEnvironment" session.

Despite these attempts, the adapter seems to default to or override with a standard Spark session.

My question to the community is:
Is it currently possible to configure dbt-fabricspark via profiles.yml to spin up a Jupyter session in Fabric?
If so, what is the correct spark_config parameter (or combination of parameters) and their expected values to achieve this?
Are there any known limitations or specific requirements on the Fabric Livy endpoint that might cause jobType or kind to be ignored or overridden?