MiniDFSCluster Docker Image

This Docker image provides a local MiniDFSCluster environment to simplify testing HDFS-based applications such as Apache Spark.

---

Features

• NameNode with fixed ports for both RPC and Web UI.
• Configurable number of DataNodes (default: 1, max: 9).
• Automatic creation of /tmp and /user directories in HDFS.
• Verified on Ubuntu, CentOS, Windows, and macOS.

---

Installation

The Docker image is available on GitHub Packages:

ghcr.io/avs-alatau/minidfscluster

Pull the Image

docker pull ghcr.io/avs-alatau/minidfscluster:latest

---

Running the Container

On Linux

docker run -d --name hdfs-minicluster --network=host \
  -e NUM_DATANODES=3 ghcr.io/avs-alatau/minidfscluster:latest

On Windows/macOS

docker run -d --name hdfs-minicluster -p 35200:35200 -p 35100:35100 -p 30000-30030:30000-30030 -e NUM_DATANODES=3 ghcr.io/avs-alatau/minidfscluster:latest

Notes

• The NUM_DATANODES environment variable is optional (default: 1).
• Maximum allowed NUM_DATANODES: 9.

---

Ports Information

Service	Port
NameNode RPC	35200
NameNode Web UI	35100
DataNode RPC	30000-30030
DataNode Transfer	30000-30030
DataNode HTTP	30000-30030

---

Usage

Checking Logs

After starting the container, wait for about 30 seconds, then check the logs:

docker logs hdfs-minicluster

This will display connection details and available ports.

Connecting to HDFS

Use the full HDFS path for operations:

hdfs dfs -ls hdfs://localhost:35200/

Pre-Created HDFS Directories

Upon startup, the following directories are created:

• /tmp
• /user

---

Example: Running with Apache Spark

You can run spark-shell and connect to HDFS:

/opt/spark-3.5.1/bin/spark-shell --master local[*] \
  --conf spark.hadoop.fs.defaultFS=hdfs://localhost:35200

---

Feedback and Contributions

For issues, improvements, or contributions, please visit the GitHub repository.

---

Happy Testing!