[R-sig-hpc] Excessive network traffic and slow code execution when running R via qsub on an HPC

[Chris Davis]

I think I managed to solve it using the code below.  With sqldf the queries were done directly on data frames without any code that initialized a connection to a sqlite database.  Somehow this led the code constantly talking to the head node.  The R code itself does repetitions of a job that are independent of each other, so the code below is run before each repetition begins.


    # close an old connection if it exists
    if (!is.null(getOption("sqldf.connection"))){
        sqldf()
    }
    # create a new connection
    sqldf()


Chris