Application compilation and execution

To submit last section’s sample application to your Spark setup you first have to compile it into a Java archive. We’ll use sbt for this purpose, but other build tools like Maven would work as well.

Install sbt on the system which you want to use for compiling. Then create a sbt build project with following file structure:

./build.sbt
./lib/spark-scala-mcsapi-connector.jar
./src/main/scala/ColumnStoreExporter.scala

The build file build.sbt contains the build information, the lib directory contains the mcsapi for Scala library as unmanaged dependency, and the src/main/scala/ directory contains the actual code to be executed.

build.sbt
name := "ColumnStoreExporter Example"

version := "1.0"

scalaVersion := "2.11.12"

libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.4.0"

To compile the project execute:

sbt package

This will build the application target/scala/columnstoreexporter-example_2.11-1.0.jar that can be copied to the Spark master to be executed.

docker cp target/scala/columnstoreexporter-example_2.11-1.0.jar SPARK_MASTER:/root
docker exec -it SPARK_MASTER spark-submit --master spark://master:7077 /root/columnstoreexporter-example_2.11-1.0.jar