[Rd] ClassNotFoundException when running distributed job using rJava package

Prof Brian Ripley ripley at stats.ox.ac.uk
Fri Feb 8 08:16:53 CET 2013


This is not the rJava support list: that is at 
http://www.rosuda.org/lists.shtml.

On 08/02/2013 06:57, Gaurav Dasgupta wrote:
> Hi,
>
> I have a MapReduce Java code, which I am calling from R using rJava. I have
> prepared the R package and tested that successfully. But when I deployed
> the package in a cluster and executed it, I am getting
> ClassNotFoundException. If I run the same job directly without integrating
> with R, it runs perfectly.
> Here is my R code:
>
> library(rJava)
> muMstSpark <- function(mesosMaster = NULL, input = NULL, output = NULL,
> scalaLib = NULL, sparkCore = NULL, inputSplits = 8) {
>    if (is.null(mesosMaster) || is.null(input) || is.null(output) ||
> is.null(scalaLib) || is.null(sparkCore)) {
>      stop("Usage: muMST(<mesosMaster>, <input>, <output>, <scalaLib>,
> <sparkCore>, [<inputSplits>]")
>    }
>
>    # Gets the absolute path of the external Scala and Java JARS
>    pkgPath = paste(system.file(package="MuMstBig"), "/jars", sep="")
>
>    # Initializes the JVM specifying the directory where the main Java class
> resides:
>    .jinit("pkgPath")
>
>    # Adds all the required JARs to the class path:
>    .jaddClassPath(paste(pkgPath, "Prims.jar", sep="/"))
>    .jaddClassPath(paste(pkgPath, "MSTInSpark.jar", sep="/"))
>    .jaddClassPath(scalaLib)
>    .jaddClassPath(sparkCore)
>
>    # Creates the R object for the main Java class:
>    obj <- .jnew("MSTInSpark")
>
>    # Calls the Java main class
>    .jcall(obj, "V", "mst", c(mesosMaster, input, output, inputSplits))
> }
> Here is the error log:
>
> 13/02/08 00:54:48 INFO cluster.TaskSetManager: Loss was due to
> java.lang.ClassNotFoundException: Prims$$anonfun$PrimsExecute$1
>   at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>   at java.lang.Class.forName0(Native Method)
>   at java.lang.Class.forName(Class.java:247)
>   at
> spark.JavaDeserializationStream$$anon$1.resolveClass(JavaSerializer.scala:20)
>   at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1574)
>   at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1495)
>   at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1731)
>   at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
>   at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1946)
>   at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1870)
>   at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752)
>   at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
>   at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1946)
>   at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1870)
>   at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752)
>   at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
>   at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350)
>   at scala.collection.immutable.$colon$colon.readObject(List.scala:435)
>   at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>   at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   at java.lang.reflect.Method.invoke(Method.java:597)
>   at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:969)
>   at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1848)
>   at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752)
>   at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
>   at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1946)
>   at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1870)
>   at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752)
>   at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
>   at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1946)
>   at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1870)
>   at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752)
>   at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
>   at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350)
>   at scala.collection.immutable.$colon$colon.readObject(List.scala:435)
>   at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>   at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   at java.lang.reflect.Method.invoke(Method.java:597)
>   at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:969)
>   at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1848)
>   at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752)
>   at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
>   at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1946)
>   at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1870)
>   at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752)
>   at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
>   at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1946)
>   at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1870)
>   at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752)
>   at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
>   at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350)
>   at spark.JavaDeserializationStream.readObject(JavaSerializer.scala:23)
>   at spark.JavaSerializerInstance.deserialize(JavaSerializer.scala:45)
>   at spark.executor.Executor$TaskRunner.run(Executor.scala:93)
>   at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>   at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>   at java.lang.Thread.run(Thread.java:662)
>
> I think R is unable to find the classpath. But I have specified that in the
> script by taking the absolute path of the JARs in the package. The
> package's installed across the cluster. Any idea, whats going wrong?
>
> Thanks,
>
> 	[[alternative HTML version deleted]]
>
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>


-- 
Brian D. Ripley,                  ripley at stats.ox.ac.uk
Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272866 (PA)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595



More information about the R-devel mailing list