[R] memory leak using XML readHTMLTable

[Duncan Temple Lang]

Hi James

  Unfortunately, I am not certain if the "latest version"
of the XML package has the garbage collection activated for the nodes.
It is quite complicated and that feature was turned off in some versions
of the package.  I suggest that you install the version of the package on github

  git at github-omg:omegahat/XML.git

I believe that will handle the garbage collection of nodes, and I'd like
to know if it doesn't.

   Best,
    D.

On 9/16/12 8:30 PM, J Toll wrote:
> Hi,
> 
> I'm using the XML package to scrape data and I'm trying to figure out
> how to eliminate the memory leak I'm currently experiencing.  In the
> searches I've done, it sounds like the existence of the leak is fairly
> well known.  What isn't as clear is exactly how to solve it.  The
> general process I'm using is this:
> 
> require(XML)
> 
> myFunction <- function(URL) {
> 
>   html <- readLines(URL)
> 
>   tables <- readHTMLTable(html, stringsAsFactors = FALSE)
> 
>   myData <- data.frame(Value = tables[[1]][, 2],
>                                 row.names = make.unique(tables[[1]][, 1]),
>                                 stringsAsFactors = FALSE)
> 
>  rm(list = c("html", "tables"))   # here, and
>  free(tables)                          # here, my attempt to solve the
> memory leak
> 
>   return(myData)
> 
> }
> 
> x <- lapply(myURLs, myFunction)
> 
> 
> I've tried using rm() and free() to try to free up the memory each
> time the function is called, but it hasn't worked as far as I can
> tell.  By the time lapply is finished woking through my list of url's,
> I'm swapping about 3GB of memory.
> 
> I've also tried using gc(), but that seems to also have no effect on
> the problem.
> 
> I'm running RStudio 0.96.330 and latest version of XML.
> R version 2.15.1 (2012-06-22) -- "Roasted Marshmallows"
> Platform: x86_64-apple-darwin9.8.0/x86_64 (64-bit)
> 
> Any suggestions on how to solve this memory issue?  Thanks.
> 
> 
> James
> 
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
> 
>