Hi,

I have a MapReduce Java code, which I am calling from R using rJava. I have
prepared the R package and tested that successfully. But when I deployed
the package in a cluster and executed it, I am getting
ClassNotFoundException. If I run the same job directly without integrating
with R, it runs perfectly.
Here is my R code:

library(rJava)
muMstSpark <- function(mesosMaster = NULL, input = NULL, output = NULL,
scalaLib = NULL, sparkCore = NULL, inputSplits = 8) {
  if (is.null(mesosMaster) || is.null(input) || is.null(output) ||
is.null(scalaLib) || is.null(sparkCore)) {
    stop("Usage: muMST(<mesosMaster>, <input>, <output>, <scalaLib>,
<sparkCore>, [<inputSplits>]")
  }

  # Gets the absolute path of the external Scala and Java JARS
  pkgPath = paste(system.file(package="MuMstBig"), "/jars", sep="")

  # Initializes the JVM specifying the directory where the main Java class
resides:
  .jinit("pkgPath")

  # Adds all the required JARs to the class path:
  .jaddClassPath(paste(pkgPath, "Prims.jar", sep="/"))
  .jaddClassPath(paste(pkgPath, "MSTInSpark.jar", sep="/"))
  .jaddClassPath(scalaLib)
  .jaddClassPath(sparkCore)

  # Creates the R object for the main Java class:
  obj <- .jnew("MSTInSpark")

  # Calls the Java main class
  .jcall(obj, "V", "mst", c(mesosMaster, input, output, inputSplits))
}
Here is the error log:

13/02/08 00:54:48 INFO cluster.TaskSetManager: Loss was due to
java.lang.ClassNotFoundException: Prims$$anonfun$PrimsExecute$1
 at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
 at java.security.AccessController.doPrivileged(Native Method)
 at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
 at java.lang.Class.forName0(Native Method)
 at java.lang.Class.forName(Class.java:247)
 at
spark.JavaDeserializationStream$$anon$1.resolveClass(JavaSerializer.scala:20)
 at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1574)
 at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1495)
 at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1731)
 at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
 at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1946)
 at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1870)
 at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752)
 at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
 at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1946)
 at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1870)
 at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752)
 at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350)
 at scala.collection.immutable.$colon$colon.readObject(List.scala:435)
 at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
 at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:969)
 at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1848)
 at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752)
 at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
 at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1946)
 at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1870)
 at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752)
 at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
 at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1946)
 at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1870)
 at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752)
 at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350)
 at scala.collection.immutable.$colon$colon.readObject(List.scala:435)
 at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
 at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:969)
 at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1848)
 at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752)
 at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
 at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1946)
 at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1870)
 at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752)
 at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
 at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1946)
 at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1870)
 at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752)
 at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350)
 at spark.JavaDeserializationStream.readObject(JavaSerializer.scala:23)
 at spark.JavaSerializerInstance.deserialize(JavaSerializer.scala:45)
 at spark.executor.Executor$TaskRunner.run(Executor.scala:93)
 at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
 at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
 at java.lang.Thread.run(Thread.java:662)

I think R is unable to find the classpath. But I have specified that in the
script by taking the absolute path of the JARs in the package. The
package's installed across the cluster. Any idea, whats going wrong?

Thanks,

	[[alternative HTML version deleted]]

