Application compilation and execution¶
To submit last section’s sample application to your Spark setup you first have to compile it into a Java archive. We’ll use sbt for this purpose, but other build tools like Maven would work as well.
Install sbt on the system which you want to use for compiling. Then create a sbt build project with following file structure:
./build.sbt
./lib/spark-scala-mcsapi-connector.jar
./src/main/scala/ColumnStoreExporter.scala
The build file build.sbt contains the build information, the lib directory contains the mcsapi for Scala library as unmanaged dependency, and the src/main/scala/ directory contains the actual code to be executed.
name := "ColumnStoreExporter Example"
version := "1.0"
scalaVersion := "2.11.12"
libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.4.0"
To compile the project execute:
sbt package
This will build the application target/scala/columnstoreexporter-example_2.11-1.0.jar
that can be copied to the Spark master to be executed.
docker cp target/scala/columnstoreexporter-example_2.11-1.0.jar SPARK_MASTER:/root
docker exec -it SPARK_MASTER spark-submit --master spark://master:7077 /root/columnstoreexporter-example_2.11-1.0.jar