I know its no consolation, but I have a similar issue with R on a MAC, also
ploting
out large numbers of raster layers.  sometimes the problem lingers even
after I clear the workspace, do gc() etc.  Almost as if R wont ask for
processor resources.

weird.

On Wed, Oct 6, 2010 at 12:06 PM, Bastien Ferland-Raymond <
bastien.ferland-raymond.1@ulaval.ca> wrote:

> Thanks a lot for your quick answer.  Here is my answer to your questions:
>
> Have you looked to see how fast your memory might be growing?
> BFR- Yes I did, it's not to bad, it's starts around 60 000ko, rise up to
> 120 000 at the most, so not too scary.
>
> Are you leaving around any large objects that should be removed?
> BFR- I was carefull making sure the function doesn't create anything that
> would be visible with objects().  Could it be creating other type (hidden)
> objects?  Maybe, but I'm not very familliar with that stuff.
>
> Have you looked to see if you are paging?
> BFR- I just red the wiki about paging, didn't know that term before.  If I
> look at perfmon, its looks like keeping steady at 6000 pages/s with rare
> peaks as high as 900 000. Does it sounds normal?  How can it affect R?
>
> Is it your CPU time that is increasing, or your wall clock time?
> BFR- If I go to the task manager - performance.  R is initially using
> around 40% of the processor (so around 80% of 1 core) but with (real) time
> passing, it gets lower and lower to get as low as 6% (12% of one core).  I
> was surprized to see that as usually my simulation in R use one whole core.
>
> It sounds like there might be some memory leak that might be causing your
> process size to grow and possibly causing paging.  You will need to gather
> some of the performance data that perfmon can provide and look at the memory
> usage, CPU time and I/O rates over time to see if there are any changes.
> BFR- The term "Memory leak" feels right with my problem.  Is there ways I
> can control/detect/prevent this kind of problem in R.  Also, how can I check
> the I/O, i never looked at that before.
>
> Thanks again
>
> Bastien
>
>
>
> On Wed, Oct 6, 2010 at 2:11 PM, Bastien Ferland-Raymond
> <bastien.ferland-raymond.1@ulaval.ca> wrote:
> > Hello R-users,
> >
> > I'm currently facing a pretty hard problem which I'm hopping you'll be
> able to help me with.  I'm using R to create images.  That alone is not the
> problem, the problem is that I'm using R to create 168 000 images...  My
> code (which is given below) use different package (raster and rgdal) to
> import a image (size 20gig) and divide it into 168 000 pictures that are 100
> pixel x 100 pixel.  The code works fine for making the images, but if I ask
> it to run all 168 000, it always breaks around 15 000.
> >
> > It starts with the code being able to make around 2 pictures per second,
> but then it slows down and after around 2000 pictures it's only 1 picture
> per second.  Later on it's getting closer to 1 pictures every 3 seconds etc.
>  until it bugs.  I have no error message, only Windows that tells me that "R
> encounter a problem and most be close..."  Initially I though it was a
> Windows problem, that I couldn't put too many file into a folder and it was
> slowering it down.  Then I divided my batch process into smaller (5000
> files) folder but it didn't help, still breaks at 15 000.  I also try to do
> gc() after each 5000 pictures to save memory but it didn't help either.  I
> removed every loops from the code because I thought it was the problem, but
> it was just faster at bugging... After the bug, I need to restart the
> computer if I want to go back to the initial speed.
> >
> > I'm pretty much running out of options.  It's there limitation in R as
> the number of files it can create in one session?  Is it a windows problem?
>  Is there better way to clear the memory than gc()? Any thought on that?
> >
> > I'm using R 2.11.1, win XP, my hard drive is NTSF, computer: intel core2
> duo E6750 32 bit with 2 gig of Ram.
> >
> > Here is my code, but I doubt it would help much with my problem:
> >
> > ########
> > # It made of 4 functions (sorry, it's french):
> >
> >
> ##########################################################################
> >
> ##########################################################################
> > ###  Ensemble des fonctions pour faire les images NDVI rouge et verte
>  ###
> >
> ##########################################################################
> > ######  Bastien Ferland-Raymond, 5 oct 2010
>  #############################
> >
> ##########################################################################
> >
> > ########
> > ## Simplement rouler le script au complet
> > ########
> > ### Library nécessaire:
> > library(raster)
> > library(rgdal)
> > library(shapefiles)
> >
> >
> #############################################################################
> > ## Fonction 1  -  NDVI a partir de coordonnee Pixel et largeur #####
> >  calculate_NDVI<- function(Type, object, VALUE) {
> >   redorgreen <- ifelse(Type=="red",2,3)
> >   list1 <- unstack(object)
> >   rast1 <- list1[[1]]
> >   rast2 <- list1[[redorgreen]]
> >   NAvalue(rast1)<- -99999
> >   NAvalue(rast2)<- -99999
> >   cells1 <-
> getValuesBlock(rast1,row=VALUE[[2]],nrow=VALUE[[3]],col=VALUE[[1]],ncol=VALUE[[3]])
> >   cells2 <-
> getValuesBlock(rast2,row=VALUE[[2]],nrow=VALUE[[3]],col=VALUE[[1]],ncol=VALUE[[3]])
> >   cells1[is.na(cells1)]<-0;
> >   cells2[is.na(cells2)]<-0;
> >   calculNDVI <-(cells1 - cells2) / (cells1 + cells2)
> >   NDVImatrix <- matrix(calculNDVI,nrow=VALUE[[3]],ncol=VALUE[[3]],
> byrow=TRUE)
> >   NDVImatrix <- NDVImatrix + 1
> >   NDVImatrix <- NDVImatrix * (255/2)
> >   return(NDVImatrix)
> >   }
> >
> #################################################################################
> > ## Fonction 1b  -  Faire le tiff
> >  make.tiff<- function(NV=newValues,TT=Type,img=imgRaster,nom){
> >  pixelNDVIMatrix <- calculate_NDVI(TT,img,NV[c(1,2,3)])
> >  newRaster <- raster(pixelNDVIMatrix)
> >  NAvalue(newRaster)<-999999
> >  nnom<-nom[NV[4]]
> >  writeRaster(newRaster,
> filename=nnom,datatype="INT1U",format="GTiff",overwrite=FALSE)
> >  aaa<-2
> > }
> >
> >
> #################################################################################
> > ## Fonction 2  -  Creation de fonction convertissant les coordonnee
> metrique en coordonnee pixels #####
> >  latlong_to_pixels<- function(Coord, facteur, meterWidth=NULL) {   #Coord
> doit être c(x,y)
> >  newX <- Coord[1] / facteur
> >  newY <- Coord[2] / facteur
> >  if(!is.null(meterWidth)){
> >   newWidth <- meterWidth / facteur
> >   return(c(newX,newY,newWidth))
> >  }
> >  return(c(newX,newY))
> >  }
> >
> >
> #############################################################################
> > ####  Fonction 3  -  Fonction principal   #####
> >  make.NDVI.photo<-
> function(tableDesPlacettes,Type,newImagesDirectory,textAndImgDirectory="U:\\kNN_Valcartier\\Photo
> aerienne"){
> >  lastWD<- getwd()
> > setwd(textAndImgDirectory)
> >  imgRaster<- stack(imageAssociee)
> >  x1<- tableDesPlacettes[,2] - xmin(imgRaster) - (tailleFenetres/2)     #
> The image origin for calculation is in the top left corner
> >  y1<- ymax(imgRaster) - tableDesPlacettes[,3] - (tailleFenetres/2)
> > coo <- cbind(x1,y1)
> >  newValues<-
> t(apply(coo,1,latlong_to_pixels,facteurMetreParPixel,tailleFenetres))
> >  newImgName<-
> paste(newImagesDirectory,substr(Type,1,3),"_","GC",tableDesPlacettes[,1],".tif",sep="")
> >
> apply(cbind(newValues,1:length(newImgName)),1,make.tiff,Type,imgRaster,nom=newImgName)
> > setwd(lastWD)
> > }
> >
> >
> > ###########################
> > ## Executing fonctions:
> > #############################
> >
> >
> > ###  loader les données brutes de fenêtres
> > fichier.fenetre.brute<-read.dbf("U:\\kNN_Valcartier\\Fenêtres 30x30 1
> octobre\\168700_30m_centroid.dbf", header=T)
> > ###  Sélectionner les fenêtres complètes
> > fenetre.complete<-round(fichier.fenetre.brute[[1]][,2],1)==900
> > ###  Sortir les centroides pour extraction
> > centro.tout.900<-fichier.fenetre.brute[[1]][fenetre.complete,c(1,5,6)]
> > #rm(fichier.fenetre.brute) ; gc()
> >
> > ## données nécessaires pour la fonction
> >  imageAssociee<- "mosaique_all_v1.img"  # nom de l'image
> >  facteurMetreParPixel<- 0.3         # combien de metre vaut un pixel
> >  tailleFenetres= 30            # en metre
> >
> >  start.time<-Sys.time();start.time
> >
> make.NDVI.photo(centro.tout.900[19137:24136,],"red","BFR\\NDVI_red_fenetre\\batch
> 3\\")
> >  stop.time<-Sys.time()
> >  time.run<-stop.time-start.time
> >  alarm()
> >  time.run
> > gc()
> >  start.time<-Sys.time();start.time
> >
> make.NDVI.photo(centro.tout.900[24137:29136,],"red","BFR\\NDVI_red_fenetre\\batch
> 4\\")
> >  stop.time<-Sys.time()
> >  time.run<-stop.time-start.time
> >  alarm()
> >  time.run
> > gc()
> >
> > #############
> >
> >
> > Voilà,
> >
> > Thanks!
> >
> > Bastien Ferland-Raymond
> > ______________________________________________
> > R-help@r-project.org mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-help
> > PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> > and provide commented, minimal, self-contained, reproducible code.
> >
>
>
>
> --
> Jim Holtman
> Cincinnati, OH
> +1 513 646 9390
>
> What is the problem that you are trying to solve?
> ______________________________________________
> R-help@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>

	[[alternative HTML version deleted]]

