[R-sig-Geo] Managing temporary files created when using the Raster package

[Jonathan Greenberg]

Hi Julie:

Glad it worked out!  Just for the sake of preserving this info on the
interwebs and to add some "tweaks" to your solutions, first, raster's
temporary files are stored in:

rasterOptions()$tmpdir

This folder can be changed to:

rasterOptions(tmpdir="/pathto/new/temp/dir")

Re: list files, if you want to full path, I use:
normalizePath(list.files(pattern=glob2rx("temp*"),full.names =TRUE))

full.names=TRUE will give you the full path to the file, although it
will be a relative path -- normalizePath "fixes" this to the absolute
path.  The pattern parameter for list.files is a "regular expression"
(http://en.wikipedia.org/wiki/Regular_expression), which many people
aren't terribly familiar with -- if you are more familiar with the
wildcard approach (your example suggests you are -- I am too), to be
safe, wrap that in a glob2rx() function so it it created properly:

glob2rx("temp*")

The pattern="temp*", since it is interpreted as a regular expression,
is not interpreted the way you probably thought it was, and when you
remove files you should be careful about that.  You could end up
deleting different files than you thought.  "*" in regular expressions
means something different than it does in wildcard patterns.

Cheers!

--j

On Sun, Mar 30, 2014 at 10:59 AM, Julie Lee-Yaw <julleeyaw at yahoo.ca> wrote:
> Hi Jonathan,
>
> Thanks for this suggestion. It took me awhile to find all the places where
> temporary rasters were being generated. After finding them, I used filename
> in all commands that were generating the files, pointing them all to a
> temporary directory and calling them all "temp_" something. I then used the
> following lines of code to remove them at the end of each loop (note
> file.remove seemed to require me to be in the directory with the files to be
> removed...)
>
> path<-"./myPermanentDirectory" # inside this directory, I had created a temp
> directory to dump temp files in
> setwd(paste(path,"/tmpdir",sep="")) # change directory to where the
> temporary files are
> do.call(file.remove,list(list.files(pattern="temp*)))
> setwd(path) # go back to permanent directory
>
> Thanks!
>
> Julie
>
>
>
> ________________________________
> From: Jonathan Greenberg <jgrn at illinois.edu>
> To: Julie Lee-Yaw <julleeyaw at yahoo.ca>
> Cc: "r-sig-geo at r-project.org" <r-sig-geo at r-project.org>
> Sent: Sunday, March 23, 2014 11:58:50 PM
> Subject: Re: [R-sig-Geo] Managing temporary files created when using the
> Raster package
>
> Julie:
>
> Why not just build a ?file.remove into your script?  Also, most raster
> functions allow you to explicitly set a filename (see e.g. ?calc -->
> the filename= parameter), rather than letting it generate a random
> filename -- this would likely make it easier for you to figure out the
> appropriate file to delete.
>
> --j
>
> On Sun, Mar 23, 2014 at 4:44 PM, Julie Lee-Yaw <julleeyaw at yahoo.ca> wrote:
>> Hi
>>
>> I am doing a lot of raster calculations repeatedly (in a loop of sorts).
>> As might be expected, this process is generating a lot of temporary files
>> and filing up my hard drive before my script finishes. The first step in my
>> script is to stack a set of very large rasters. This stack is then the input
>> for the function that I'm using (e.g. the loop). Can I add in a line in my
>> loop (e.g. removeTmpFiles) to delete temporary files created during the loop
>> without ruining the integrity of the raster stack?
>>
>> For example:
>>
>> files<-list.files(pattern=".tif")
>> myStack<-stack(files)
>>
>> myFunction<-function(x, rasterStack, y, z){
>> # several calculations using the raster stack and x, y, z
>> # non-raster results from the calculations get saved to file using
>> write.csv()
>> removeTmpFiles(h=0) # remove temp. raster files from disk???
>> }
>>
>> lapply(x=list_of_x_values, myFunction, rasterStack=myStack, y=something,
>> z=somethingelse)
>>
>> Is the removeTmpFiles line appropriate here?
>>
>> Thanks for the help!
>
>>        [[alternative HTML version deleted]]
>>
>>
>> _______________________________________________
>> R-sig-Geo mailing list
>> R-sig-Geo at r-project.org
>> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>>
>
>
>
> --
> Jonathan A. Greenberg, PhD
> Assistant Professor
> Global Environmental Analysis and Remote Sensing (GEARS) Laboratory
> Department of Geography and Geographic Information Science
> University of Illinois at Urbana-Champaign
> 259 Computing Applications Building, MC-150
> 605 East Springfield Avenue
> Champaign, IL  61820-6371
> Phone: 217-300-1924
> http://www.geog.illinois.edu/~jgrn/
> AIM: jgrn307, MSN: jgrn307 at hotmail.com, Gchat: jgrn307, Skype: jgrn3007
>



-- 
Jonathan A. Greenberg, PhD
Assistant Professor
Global Environmental Analysis and Remote Sensing (GEARS) Laboratory
Department of Geography and Geographic Information Science
University of Illinois at Urbana-Champaign
259 Computing Applications Building, MC-150
605 East Springfield Avenue
Champaign, IL  61820-6371
Phone: 217-300-1924
http://www.geog.illinois.edu/~jgrn/
AIM: jgrn307, MSN: jgrn307 at hotmail.com, Gchat: jgrn307, Skype: jgrn3007