You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Any examples - in particular on using RunOnSpark.runOnSpark() ? #7
I can not find any documentation on the web for .ssc files. They contain scala, java, python, and some custom language. Please point some references about them.
From what I understand, the .ssc files are just a custom extension that we use to denote a scala script e.g. you can load them into a scala interpreter using scala -i test.ssc. The spark shell also used to support the -i command, though I don't think it has been working since 1.5.0+. I've been using :load path_to_file.ssc inside the Spark REPL.
I don't know about the .ssc files that aren't in scala, but I imagine it'd be for a similar purpose? @jcanny
the obvious problem is that some of the setup stuff like opts variable that are created via init_bidmach.scala are not there: so it fails. What is the way to run this in REPL?
You can delete the MockLearner import or pull from BIDMach_Spark. It's a remnant from an older version of the test.
Also, you'll have to specify the MASTER_DNS as the master DNS address of your spark machine, and also supply your own data in BIDMat matrix forms.
For example, I use the mnist8M dataset which can be downloaded using BIDMach/scripts/getmnist8m.sh. The script automatically saves the matrices into FMats. You can then run BIDMach_Spark/scripts/load_mnist.ssc and BIDMach_Spark/scripts/append_mnist.ssc to load the data onto HDFS.
How do we load the jars? They are not complementary:
scala> :require /git/BIDMat/BIDMat.jar
Added '/git/BIDMat/BIDMat.jar' to classpath.
scala> :require /git/BIDMach/BIDMach.jar
The path '/git/BIDMach/BIDMach.jar' cannot be loaded, because existing classpath entries conflict.
But if I just do the BIDMach.jar we get this:
error: missing or invalid dependency detected while loading class file 'MatIO.class'.
Could not access type MatIOtrait in package BIDMat,
because it (or its dependencies) are missing. Check your build definition for
missing or conflicting dependencies. (Re-run with -Ylog-classpath to see the problematic classpath.)
You can use the --jars option with a comma separated list of jars. Alternatively, you can configure the spark conf via spark.driver.extraClassPath and spark.executor.extraClassPath.
Hi James, yes but that was not my question/issue here. If you review again my last post it was about : "which" jars need to be used - given the apparent incompatibility of the combinations I tried.
Please say explicitly which combination of jars works for you - i.e. that you are using to get this to work.
This is a significant task since the native Hadoop binaries are not installed by default. Do you have a suggestion on how to approach this if lz4 were not available? Otherwise this is going to take me some time to understand how to install on my mac.
Apparently the Mac brew uses MH_DYLD flags and not MH_BUNDLE - contributing to my challenges here. I am kinda stalled now: setting up native library builds is not within my prior experience. I would suggest that it were likely I will not be the only one running into this issue.
Sorry I am not familiar with that. My mac installation of hadoop has the required native library (libhadoop.so), but I haven't run BIDMach_Spark on my mac before. I'm in Switzerland right now so I'll be sleeping soon, but I can try running through things on my mac tomorrow if you haven't figured it out.
Thanks James . I will update here in a bit: presently trying to build lz4 natively but the Makefile does not mention either MH_DYLD or MH_BUNDLE. This is new territory for me.
I downloaded and built hadoop with native to obtain libhadoop.dynlib. However "hadoop checknative" returns all false. I am still struggling on my end with this.
I needed a day off to attend to other things: got back on this earlier today. I fixed the hadoop native libs issue: it had to do with setting the DYLD_LIBRARY_PATH.
After that - still many things to do for this to work. Setting spark and hadoop settings, and creating a smaller version of the mnist files (8M too big for my machine). It seems to be going now on a small slice of the data.
Activity
jamesjia94 commentedon Jun 11, 2016
I created a KMeans learner that uses RunOnSpark under BIDMach_Spark/scripts/KMeansLearner.ssc.
Let me know if you have any questions!
javadba commentedon Jun 11, 2016
Sounds perfect! Taking a look.
javadba commentedon Jun 11, 2016
How do I run it? In general : is there a reference about .ssc files? I do not undertand: where is the interpreter for them and how they work.
javadba commentedon Jun 11, 2016
I can not find any documentation on the web for .ssc files. They contain scala, java, python, and some custom language. Please point some references about them.
jamesjia94 commentedon Jun 11, 2016
From what I understand, the .ssc files are just a custom extension that we use to denote a scala script e.g. you can load them into a scala interpreter using scala -i test.ssc. The spark shell also used to support the -i command, though I don't think it has been working since 1.5.0+. I've been using :load path_to_file.ssc inside the Spark REPL.
I don't know about the .ssc files that aren't in scala, but I imagine it'd be for a similar purpose? @jcanny
javadba commentedon Jun 11, 2016
So when I do
the obvious problem is that some of the setup stuff like
optsvariable that are created via init_bidmach.scala are not there: so it fails. What is the way to run this in REPL?jamesjia94 commentedon Jun 11, 2016
KMeans.learner() returns a tuple of (learner, opts), so opts should be in scope. Could you paste the specific error that you are receiving?
javadba commentedon Jun 11, 2016
How are you starting spark-shell? The BIDMat/Mach libraries will need to be present.
Here is what I have done so far:
cala> :require /git/BIDMach/BIDMach.jar
Added '/git/BIDMach/BIDMach.jar' to classpath.
scala> :require /git/BIDMach_Spark/BIDMach_Spark.jar
Cannot load '/git/BIDMach_Spark/BIDMach_Spark.jar'
scala> :require /git/BIDMach_Spark/BIDMatHDFS.jar.jar
Cannot load '/git/BIDMach_Spark/BIDMatHDFS.jar.jar'
scala> :require /git/BIDMach_Spark/BIDMatHDFS.jar
Added '/git/BIDMach_Spark/BIDMatHDFS.jar' to classpath.
scala> :load /git/BIDMach_Spark/scripts/KMeansLearner.ssc
Loading /git/BIDMach_Spark/scripts/KMeansLearner.ssc...
import org.apache.spark.rdd.RDD
import scala.reflect.ClassTag
import BIDMat.MatIO
import BIDMat.SerText
import BIDMach.RunOnSpark._
:30: error: object MockLearner is not a member of package BIDMach
import BIDMach.MockLearner
^
jamesjia94 commentedon Jun 12, 2016
You can delete the MockLearner import or pull from BIDMach_Spark. It's a remnant from an older version of the test.
Also, you'll have to specify the MASTER_DNS as the master DNS address of your spark machine, and also supply your own data in BIDMat matrix forms.
For example, I use the mnist8M dataset which can be downloaded using BIDMach/scripts/getmnist8m.sh. The script automatically saves the matrices into FMats. You can then run BIDMach_Spark/scripts/load_mnist.ssc and BIDMach_Spark/scripts/append_mnist.ssc to load the data onto HDFS.
javadba commentedon Jun 12, 2016
OK I pulled your update .
How do we load the jars? They are not complementary:
scala> :require /git/BIDMat/BIDMat.jar
Added '/git/BIDMat/BIDMat.jar' to classpath.
scala> :require /git/BIDMach/BIDMach.jar
The path '/git/BIDMach/BIDMach.jar' cannot be loaded, because existing classpath entries conflict.
But if I just do the BIDMach.jar we get this:
error: missing or invalid dependency detected while loading class file 'MatIO.class'.
Could not access type MatIOtrait in package BIDMat,
because it (or its dependencies) are missing. Check your build definition for
missing or conflicting dependencies. (Re-run with
-Ylog-classpathto see the problematic classpath.)jamesjia94 commentedon Jun 13, 2016
You can use the --jars option with a comma separated list of jars. Alternatively, you can configure the spark conf via spark.driver.extraClassPath and spark.executor.extraClassPath.
javadba commentedon Jun 13, 2016
Hi James, yes but that was not my question/issue here. If you review again my last post it was about : "which" jars need to be used - given the apparent incompatibility of the combinations I tried.
Please say explicitly which combination of jars works for you - i.e. that you are using to get this to work.
jamesjia94 commentedon Jun 13, 2016
If you use --jars BIDMach.jar,BIDMat.jar instead of :require, the scala/Spark repl doesn't throw any errors.
That being said, here is my spark conf.
spark.executor.extraClassPath /opt/BIDMach/lib/BIDMatHDFS.jar:/opt/BIDMach/BIDMach.jar:/opt/BIDMach/lib/BIDMat.jar:/opt/BIDMach/lib/ptplot.jar:/opt/BIDMach/lib/ptplotapplication.jar:/opt/BIDMach/lib/jhdf5.jar:/opt/BIDMach/lib/commons-math3-3.2.jar:/opt/BIDMach/lib/lz4-1.3.jar:/opt/BIDMach/lib/json-io-4.1.6.jar:/opt/BIDMach/lib/jfreechart-1.0.19.jar:/opt/BIDMach/lib/jcommon-1.0.23.jar:/opt/BIDMach/lib/jcuda-0.7.0a.jar:/opt/BIDMach/lib/jcublas-0.7.0a.jar:/opt/BIDMach/lib/jcufft-0.7.0a.jar:/opt/BIDMach/lib/jcurand-0.7.0a.jar:/opt/BIDMach/lib/jcusparse-0.7.0a.jar
spark.driver.extraClassPath /opt/BIDMach/lib/BIDMatHDFS.jar:/opt/BIDMach/BIDMach.jar:/opt/BIDMach/lib/BIDMat.jar:/opt/BIDMach/lib/ptplot.jar:/opt/BIDMach/lib/ptplotapplication.jar:/opt/BIDMach/lib/jhdf5.jar:/opt/BIDMach/lib/commons-math3-3.2.jar:/opt/BIDMach/lib/lz4-1.3.jar:/opt/BIDMach/lib/json-io-4.1.6.jar:/opt/BIDMach/lib/jfreechart-1.0.19.jar:/opt/BIDMach/lib/jcommon-1.0.23.jar:/opt/BIDMach/lib/jcuda-0.7.0a.jar:/opt/BIDMach/lib/jcublas-0.7.0a.jar:/opt/BIDMach/lib/jcufft-0.7.0a.jar:/opt/BIDMach/lib/jcurand-0.7.0a.jar:/opt/BIDMach/lib/jcusparse-0.7.0a.jar
spark.driver.extraLibraryPath /opt/BIDMach/lib:/usr/local/cuda/lib64:/usr/local/hadoop/lib/native
spark.executor.extraLibraryPath /opt/BIDMach/lib:/usr/local/cuda/lib64:/usr/local/hadoop/lib/native
javadba commentedon Jun 13, 2016
Thanks a lot James - I did not realize that ":require" would behave differently than --jars. Trying now.
javadba commentedon Jun 13, 2016
How did you convert the MNIST files to Sequence files?
javadba commentedon Jun 13, 2016
Oh right the HDFSIO class. Give me few mins on that.
javadba commentedon Jun 13, 2016
Looks like your load_mnist.ssc should do the trick.
jamesjia94 commentedon Jun 13, 2016
Yes, that's correct. You can then merge all those files into one file via append_mnist.ssc to create a larger dataset.
javadba commentedon Jun 13, 2016
thx - yes I did see that too. Few mins should have something here.
javadba commentedon Jun 13, 2016
How did you set the lz4? I have been unable to get past the following:
java.lang.RuntimeException: native lz4 library not available
jamesjia94 commentedon Jun 13, 2016
Did you add the path to hadoop native libraries to the library path of your executors and driver?
javadba commentedon Jun 13, 2016
The lz4 is not residing under hadoop in my case. I did add the path to the lz4 under SPARK_LIBRARY_PATH in $SPARK_HOME/conf/spark-env.sh
javadba commentedon Jun 13, 2016
oh ok - so the problem may be due to Hadoop not having access to lz4 - not spark. I am going to add lz4 to hadoop now
javadba commentedon Jun 13, 2016
This is a significant task since the native Hadoop binaries are not installed by default. Do you have a suggestion on how to approach this if lz4 were not available? Otherwise this is going to take me some time to understand how to install on my mac.
javadba commentedon Jun 13, 2016
Apparently the Mac brew uses MH_DYLD flags and not MH_BUNDLE - contributing to my challenges here. I am kinda stalled now: setting up native library builds is not within my prior experience. I would suggest that it were likely I will not be the only one running into this issue.
jamesjia94 commentedon Jun 13, 2016
Sorry I am not familiar with that. My mac installation of hadoop has the required native library (libhadoop.so), but I haven't run BIDMach_Spark on my mac before. I'm in Switzerland right now so I'll be sleeping soon, but I can try running through things on my mac tomorrow if you haven't figured it out.
javadba commentedon Jun 13, 2016
Thanks James . I will update here in a bit: presently trying to build lz4 natively but the Makefile does not mention either MH_DYLD or MH_BUNDLE. This is new territory for me.
javadba commentedon Jun 14, 2016
I downloaded and built hadoop with native to obtain libhadoop.dynlib. However "hadoop checknative" returns all false. I am still struggling on my end with this.
javadba commentedon Jun 16, 2016
I needed a day off to attend to other things: got back on this earlier today. I fixed the hadoop native libs issue: it had to do with setting the DYLD_LIBRARY_PATH.
After that - still many things to do for this to work. Setting spark and hadoop settings, and creating a smaller version of the mnist files (8M too big for my machine). It seems to be going now on a small slice of the data.
I will put together a PR tomorrow.
javadba commentedon Jun 16, 2016
PR is in #8