Spark on Solaris - Codec Problem

In my project, I need to run Apache Spark on Solaris like Hadoop :) (in my previous blog entry)

While executing my application (even sample of Apache spark), I have faced the following error ;

Exception in thread "main" java.lang.IllegalArgumentException

at org.apache.spark.io.SnappyCompressionCodec.<init>(CompressionCodec.scala:152)

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

at java.lang.reflect.Constructor.newInstance(Constructor.java:526)

at org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:68)

at org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:60)

at org.apache.spark.broadcast.TorrentBroadcast.org$apache$spark$broadcast$TorrentBroadcast$$setConf(TorrentBroadcast.scala:73)

at org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBroadcast.scala:80)

at org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34)

at org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:62)

at org.apache.spark.SparkContext.broadcast(SparkContext.scala:1289)

at org.apache.spark.SparkContext$$anonfun$hadoopFile$1.apply(SparkContext.scala:977)

at org.apache.spark.SparkContext$$anonfun$hadoopFile$1.apply(SparkContext.scala:974)

at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:148)

at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:109)

at org.apache.spark.SparkContext.withScope(SparkContext.scala:681)

at org.apache.spark.SparkContext.hadoopFile(SparkContext.scala:974)

at org.apache.spark.SparkContext$$anonfun$textFile$1.apply(SparkContext.scala:799)

at org.apache.spark.SparkContext$$anonfun$textFile$1.apply(SparkContext.scala:797)

at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:148)

at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:109)

at org.apache.spark.SparkContext.withScope(SparkContext.scala:681)

at org.apache.spark.SparkContext.textFile(SparkContext.scala:797)

at org.apache.spark.api.java.JavaSparkContext.textFile(JavaSparkContext.scala:184)

at com.ericsson.experimental.spark.jobs.Collaborative.collabFilter(Collaborative.java:162)

at com.ericsson.experimental.spark.jobs.Collaborative.main(Collaborative.java:110)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:664)

at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:169)

at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:192)

at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:111)

at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

 

Its root cause is not having Snappy codec on solaris. It is possible to install Snappy codec by following the instructions.

But I couldn't make it working ... So I deciced to switch LZ4 codec by the following parameter.

/usr/local/spark-1.4.0/bin/./spark-submit --conf spark.io.compression.codec=lz4 --class com.sample.experimental.spark.jobs.CollaborativeSample ~/spark-sample-0.0.1-SNAPSHOT.jar hdfs://sparksrv1:19000/in/data