Hi,

 

I'm using R to collect data for a number of exchanges through a socket
connection and constantly running into memory problems even though task I
believe is not that memory consuming. I guess there is a miscommunication
between R and WinXP about freeing up memory.

So this is the code:

 

for (x in 1:length(exchanges.to.get)) {

   tickers<-sqlQuery(channel,paste("SELECT Symbol FROM symbols_list WHERE
Exchange='",exchanges.to.get[x],"';",sep=''))[,1]

   dir.create(paste(Working.dir,exchanges.to.get[x],'/',sep=''))

   for (y in 1:length(tickers)) {

     con2 <- socketConnection(Sys.info()["nodename"], port = ****)  #open
socket connection to get data

     writeLines(paste(command,',',tickers[y],',',interval,';',sep=''), con2)

     data.<-readLines(con2)

     end.of.data<-sum(c(data.=="!ENDMSG!",data.=="!SYNTAX_ERROR!"))

     while(end.of.data!=1)
{new.data<-readLines(con2);end.of.data<-sum(new.data=="!ENDMSG!");
data.<-c(data.,new.data)}

     if (length(data.)>3)
write.table(data.[1:(length(data.)-2)],paste(Working.dir,exchanges.to.get[x]
,'/',sub('\\*','\+',tickers[y]),'_.csv',sep=''),quote=F,col.names =
F,row.names=F)

     close(con2)

   }

  rm(tickers)

  gc()

 

 

With command  gcinfo(TRUE) I got the following info (some examples) :

 

Garbage collection 16362 = 15411+754+197 (level 0) ... 

6.3 Mbytes of cons cells used (22%)

2.2 Mbytes of vectors used (8%)

 

Garbage collection 16407 = 15454+756+197 (level 0) ... 

13.1 Mbytes of cons cells used (46%)

10.4 Mbytes of vectors used (39%)

 

Garbage collection 16410 = 15456+756+198 (level 2) ... 

4.9 Mbytes of cons cells used (21%)

0.9 Mbytes of vectors used (4%)

 

Garbage collection 16679 = 15634+796+249 (level 0) ... 

150.7 Mbytes of cons cells used (95%)

203.9 Mbytes of vectors used (75%)

 

Garbage collection 16680 = 15634+796+250 (level 2) ... 

4.9 Mbytes of cons cells used (4%)

0.9 Mbytes of vectors used (0%)

 

Garbage collection 16808 = 15754+802+252 (level 0) ... 

6.1 Mbytes of cons cells used (7%)

1.8 Mbytes of vectors used (1%)

 

But the end result is in Task Manager:

RGui.exe  Mem Usage 470,472K  VM Size 541,988K

 

Even though R reports 

Garbage collection 16808 = 15754+802+252 (level 0) ... 

6.1 Mbytes of cons cells used (7%)

1.8 Mbytes of vectors used (1%)

 

Has anybody encountered this problem and how you guys deal with it?  It
seems like a memory leak to me, as tasks are not memory demandind, the
biggest amount of data in a single file is about 40MB.

 

Thanks


	[[alternative HTML version deleted]]

