Run and Debug Dotnet Spark application on Docker¶
Warning
Please go to "Running Dotnet Spark applications on Ubuntu Continer" for running dotnet spark applicatoin on Docker
Instal Docker Desktop¶
Tip
Make sure to restart the machine after installation. If there is an access issue add current user to docker group.
User Docker container¶
Precompiled Container If you want to custom compile, visit Dotnet Spark Container Script and build.sh
docker run -d --name dotnet-spark -p 8080:8080 -p 8081:8081 -e SPARK_DEBUG_DISABLED=true 3rdman/dotnet-spark:latest
Docker container should be running as
Builing and exectuting the example application¶
Go to docker bash
Navigate to the project
Where json file as sample data. Now let's build
Once the build is completed, there will be bin and obj folders and let's navigate to the bin folder now
The sample application expects json file, so let's copy it from the application folder
Finally, let's execute the application spark-submit
spark-submit --class org.apache.spark.deploy.dotnet.DotnetRunner --master spark://$HOSTNAME:$SPARK_MASTER_PORT microsoft-spark-2.4.x-0.12.1.jar dotnet HelloSpark.dll
Tip
Have a proper version of jar from the bin folder i.e microsoft-spark-2.4.x-0.12.1.jar
We can verify the application and workder at http://localhost:8080/ 
Debug Spark applciation using VS Code¶
Start the container with mounted project volume or folder
docker run -d --name dotnet-spark -p 8080:8080 -p 8081:8081 -v "C:\Projects\ML\mySparkApp\bin\Debug:/dotnet/Debug" -e SPARK_DEBUG_DISABLED=true 3rdman/dotnet-spark:latest
Unfortunately, there is an exception due to a lot more different undefined dyanmic port mapping there 
This can be resolve in linus using host network as
docker run -d --name dotnet-spark --network host -v "$HOME/Projects/ML/mySparkApp/bin/Debug:/dotnet/Debug" -e SPARK_DEBUG_DISABLED=true 3rdman/dotnet-spark:latest
Warning
So we might get fix for window from Spark or Docker in near future till then stick with Linux