Azure Data Lake Store Support

[strtdusty]

Would you be interested in a PR adding support for communicating with Azure Datalake?

The changes I have made to enable it locally:

• Switch the spark package to use "without-hadoop"
• Download the full hadoop dist.
• Upgrade Hadoop version to 3.2.2
• Add a couple of environment variables for HADOOP
• Add the necessary hadoop/azure jar files to $SPARK_HOME/jars

I am specifically interested in having it in the runtime image but could see its use in the interactive as well.

[strtdusty]

Minimally, how about at least a PR to rev the version of HADOOP to something greater than 2.7 (3.2 would be the best I think)? I think I could handle the remainder with spark-submit package parameters.

[indy-3rdman]

@strtdusty, I've added a new image type runtime-hadoop that comes with a full installation of hadoop.
You should be able to try it out via docker pull 3rdman/dotnet-spark:2.1.1-3.2.1-3.2.3-hadoop for example.

Could you please give it a try and let me know whether it works as expected?