spark-tree-plotting lib seems to have lost support for Spark 2.2+

[michalrudko]

Hi,

We were happy to find this lib as we need to visualize the decision tree created using ML, but we are afraid that it cannot support the current Spark version.
We were trying multiple options in terms of setup, but when calling the plot_tree function, we get the following error:

ERROR:root:Exception while sending command.
Traceback (most recent call last):
File "/opt/cloudera/parcels/SPARK2/lib/spark2/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py", line 883, in send_command
response = connection.send_command(command)
File "/opt/cloudera/parcels/SPARK2/lib/spark2/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py", line 1040, in send_command
"Error while receiving", e, proto.ERROR_ON_RECEIVE)
Py4JNetworkError: Error while receiving

(...)

Py4JError: com.jasoto.spark.ml.SparkMLTree does not exist in the JVM

Code:

SUBMIT_ARGS = "--packages julioasotodv:spark-tree-plotting:0.2 pyspark-shell"
os.environ["PYSPARK_SUBMIT_ARGS"] = SUBMIT_ARGS

spark = (
SparkSession
.builder
.appName('jartest')
#.config('spark.jars', '/tmp/UniCredit_T100900/jars/spark-tree-plotting-0.2.jar')
.getOrCreate()
)

We'd appreciate some support on usage of this package.

Regards,
Michal

[witwall]

try in this way, I have tested it on spark2.3.1 and spark 2.4, both works

from pyspark.sql import SparkSession
spark = SparkSession.builder.\
config("spark.jars","/Users/steven/spark/spark-tree-plotting/*").\
getOrCreate()

[cerenaaa]

I get the same error when I run:

from spark_tree_plotting import plot_tree # requires pydot3

your_dtree_classification_model = dtModel

png_string = plot_tree(dtModel,
featureNames=features,
categoryNames=[],
classNames=[],
filled=True,
roundedCorners=True,
roundLeaves=True)

on Databricks. Is this not compatible with Databricks?

Thanks!

[jdongca2003]

I created the pull request to solve this issue: #7
You can check out this forked copy : https://github.com/jdongca2003/spark-tree-plotting

Here I used SBT 1.6.2 Run : sbt assembly and load the assembly jar into Databricks cluster library.

[Arnold1]

@wlbhiro @cerenaaa @jdongca2003 @agupta004 @witwall any idea how to use that jar file in Databricks?
I created the jar file with:

$ git config --get remote.origin.url
https://github.com/jdongca2003/spark-tree-plotting

$ git branch
* master
$ git log
commit b3bc0f7a6d2aa977527609b6405472fd1f17693e (HEAD -> master, origin/master, origin/HEAD)
Author: JIANXIONG DONG <jdongca2003@gmail.com>
Date:   Fri Feb 11 19:25:17 2022 -0800

    package the python scripts based on standard pip package

$ sbt version
[info] welcome to sbt 1.6.2 (Oracle Corporation Java 1.8.0_231)
[info] loading settings for project spark-tree-plotting-build from assembly.sbt ...
[info] loading project definition from /Users/arnold1/spark_plotting_test/spark-tree-plotting/project
[info] loading settings for project spark-tree-plotting from build.sbt ...
[info] set current project to spark-tree-plotting (in build file:/Users/arnold1/spark_plotting_test/spark-tree-plotting/)
[info] 0.3

$ sbt clean; sbt compile; sbt assembly
[info] welcome to sbt 1.6.2 (Oracle Corporation Java 1.8.0_231)
[info] loading settings for project spark-tree-plotting-build from assembly.sbt ...
[info] loading project definition from /Users/arnold1/spark_plotting_test/spark-tree-plotting/project
[info] loading settings for project spark-tree-plotting from build.sbt ...
[info] set current project to spark-tree-plotting (in build file:/Users/arnold1/spark_plotting_test/spark-tree-plotting/)
[success] Total time: 1 s, completed Jun 1, 2022 10:56:37 PM
[info] welcome to sbt 1.6.2 (Oracle Corporation Java 1.8.0_231)
[info] loading settings for project spark-tree-plotting-build from assembly.sbt ...
[info] loading project definition from /Users/arnold1/spark_plotting_test/spark-tree-plotting/project
[info] loading settings for project spark-tree-plotting from build.sbt ...
[info] set current project to spark-tree-plotting (in build file:/Users/arnold1/spark_plotting_test/spark-tree-plotting/)
[info] Executing in batch mode. For better performance use sbt's shell
[info] compiling 1 Scala source to /Users/arnold1/spark_plotting_test/spark-tree-plotting/target/scala-2.12/classes ...
[warn] there was one feature warning; re-run with -feature for details
[warn] one warning found
[success] Total time: 17 s, completed Jun 1, 2022 10:57:06 PM
[info] welcome to sbt 1.6.2 (Oracle Corporation Java 1.8.0_231)
[info] loading settings for project spark-tree-plotting-build from assembly.sbt ...
[info] loading project definition from /Users/arnold1/spark_plotting_test/spark-tree-plotting/project
[info] loading settings for project spark-tree-plotting from build.sbt ...
[info] set current project to spark-tree-plotting (in build file:/Users/arnold1/spark_plotting_test/spark-tree-plotting/)
[info] Strategy 'discard' was applied to a file (Run the task at debug level to see details)
[success] Total time: 3 s, completed Jun 1, 2022 10:57:17 PM

than imported that jar file in Databricks (https://docs.databricks.com/libraries/workspace-libraries.html).

but still get this error: ModuleNotFoundError: No module named 'spark_tree_plotting' when I try to import:

from spark_tree_plotting import plot_tree

[witwall]

give up this one, and have a look https://github.com/parrt/dtreeviz,

intro

A python library for decision tree visualization and model interpretation. Currently supports scikit-learn, XGBoost, Spark MLlib, and LightGBM trees.

some samples