ADH-7037: spark4 engine integration

[werzerbb]

Summary

Adds Spark 4.2 as a second, coexisting engine alongside Spark 3.5: kyuubi-server stays Spark-version-neutral.
Spark engine received a -Pspark-4.2 build profile. A client can select which engine version/profile to launch per session over Spark Connect, and build/dist can package engine jars for multiple Spark versions into a single distribution (--extra-spark-profiles parameter).

What was added

• Spark 4.2 engine profile: the changes are copied from
  upstream (Support Spark 4.2 apache/kyuubi#7570);
• Ranger authz support for Spark 4.0/4.1: copied from upstream (Authz supports Spark 4.0 and 4.1 apache/kyuubi#7552).
• Per-Spark-version artifact IDs: kyuubi-spark-sql-engine is now built as
  kyuubi-spark-sql-engine-spark-${spark.artifact.version}_${scala.binary.version}, so 3.5 and 4.2
  engine jars can coexist in the same distribution without overwriting each other. Same suffix
  pattern applied to kyuubi-spark-connect-client/kyuubi-spark-connect-client-standalone.
• Client-selected engine profile over Spark Connect: a Spark Connect client can request specific engine profile (e.g. spark4) via the connection string (kyuubi.engine.profile=spark4), letting one Kyuubi server route different sessions to different Spark major versions' engines.
• build/dist --extra-spark-profiles: comma-separated list of additional Spark profiles (e.g. spark-4.2) to also build the engine jar for, after the primary -Pspark-X.Y build.
• Python client (kyuubi/spark_connect.py): KyuubiSessionBuilder updated to work against both Spark 3.5's and Spark 4.x's differing ChannelBuilder/DefaultChannelBuilder APIs.
• JVM Spark Connect client made Spark-4-compatible: KyuubiSessionBuilder.getOrCreate() (extensions/spark/kyuubi-spark-connect-client) now resolves the concrete SparkSession class and calls builder()/client()/getOrCreate() via DynClasses/ReflectUtils.invokeAs instead of a direct compiled call.

Build

export JAVA_HOME=/usr/lib/jvm/java-21-openjdk-amd64
build/dist --tgz --spark-provided --flink-provided --hive-provided --web-ui \
  --extra-spark-profiles spark-4.2 \
  -Pjdbc-shaded -Pscala-2.13 -Pspark-3.5 -Pzookeeper-3.9 -Drat.skip=true

This produces one tarball with engine jars for both -Pspark-3.5 (primary) and -Pspark-4.2
(extra), under externals/engines/spark/.

Examples (running pyspark via Spark Connect)

# Spark 4.2 engine
KYUUBI_AUTH=kerberos pyspark4 --remote "sc://vdmitriev-adh-sun2-hadoop-1.ru-central1.internal:2181,vdmitriev-adh-sun2-hadoop-2.ru-central1.internal:2181,vdmitriev-adh-sun2-hadoop-3.ru-central1.internal:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=arenadata/cluster/11/kyuubi_sc;use_ssl=true;kyuubi.engine.profile=spark4"

# Spark 3.5 engine - same command, pyspark3 + kyuubi.engine.profile=spark3
KYUUBI_AUTH=kerberos pyspark3 --remote "sc://vdmitriev-adh-sun2-hadoop-1.ru-central1.internal:2181,vdmitriev-adh-sun2-hadoop-2.ru-central1.internal:2181,vdmitriev-adh-sun2-hadoop-3.ru-central1.internal:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=arenadata/cluster/11/kyuubi_sc;use_ssl=true;kyuubi.engine.profile=spark3"