Skip to content

[feature][linkis][engineplugin]the spark engine supports setting executor confs#937

Merged
casionone merged 7 commits intodev-1.18.1-webankfrom
feature/spark-driver-params
Mar 13, 2026
Merged

[feature][linkis][engineplugin]the spark engine supports setting executor confs#937
casionone merged 7 commits intodev-1.18.1-webankfrom
feature/spark-driver-params

Conversation

@v-kkhuang
Copy link

@v-kkhuang v-kkhuang commented Mar 12, 2026 •
edited
Loading

What is the purpose of the change

This PR adds support for dynamically setting Spark driver parameters in Spark EngineConn with configuration control, Spark version verification, and exception isolation to ensure task execution stability.

Related issues/PRs

Related issues: close #590
Related pr:

Brief change log

  • Add configuration wds.linkis.spark.driver.params.enabled to control the feature (default: false).
  • Add configuration wds.linkis.spark.driver.params.exclude to exclude parameters (comma-separated).
  • Apply Spark driver parameters after sc.setJobGroup in the executeLine method.
  • Add Spark version check: only execute for Spark 3.4.4 and above.
  • Add exception isolation: log WARNING when parameter setting fails without interrupting task.

Checklist

  • I have read the Contributing Guidelines on pull requests.
  • I have explained the need for this PR and the problem it solves.
  • I have explained the changes or the new features added to this PR.
  • I have added tests corresponding to this change.
  • I have updated the documentation to reflect this change.
  • I have verified that this change is backward compatible.
  • If this is a code change: I have written unit tests to fully verify the new behavior.

@v-kkhuang @claude
#AI commit#开发阶段：新增功能 - spark引擎支持设置driver参数到driver端
36a1d0f 
功能：
1. 新增配置项 wds.linkis.spark.driver.params.enabled 控制功能开关（默认关闭）
2. 新增配置项 wds.linkis.spark.driver.params.exclude 排除不设置的参数（逗号分隔）
3. 在executeLine方法中sc.setJobGroup后执行参数设置
4. 支持Spark版本检查，仅在3.4.4及以上版本执行
5. 异常隔离：参数设置失败记录WARNING日志，不影响任务执行

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@v-kkhuang v-kkhuang changed the title spark引擎支持设置driver参数到driver端 [linkis][engineplugin]the Spark engine supports setting driver parameters on the driver side. Mar 12, 2026
@v-kkhuang v-kkhuang changed the title [linkis][engineplugin]the Spark engine supports setting driver parameters on the driver side. [linkis][engineplugin]the spark engine supports setting driver parameters on the driver side. Mar 12, 2026
casionone
casionone approved these changes Mar 13, 2026
View reviewed changes
Copy link

@casionone casionone left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@casionone casionone changed the title [linkis][engineplugin]the spark engine supports setting driver parameters on the driver side. [linkis][engineplugin]the spark engine supports setting executor confs Mar 13, 2026
@casionone casionone changed the title [linkis][engineplugin]the spark engine supports setting executor confs [feature][linkis][engineplugin]the spark engine supports setting executor confs Mar 13, 2026
@casionone casionone added the enhancement New feature or request label Mar 13, 2026
@casionone casionone merged commit c54650b into dev-1.18.1-webank Mar 13, 2026
12 of 16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@casionone casionone casionone approved these changes

Assignees

No one assigned

Labels

enhancement New feature or request

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@v-kkhuang @casionone