<html><head></head><body><div style="font-family: Verdana;font-size: 12.0px;"><div>Jonathan,<br/></div><div><br/></div><div>On some filesystems (e.g. NTFS, see below) it is possible to create 'sparse' memory-mapped files, i.e. reserving the space without the cost of actually writing initial values. <br/></div><div class="signature">Package 'ff' does this automatically and also allows to access the file in parallel. Check the example below and see how big file creation is immediate.<br/></div><div class="signature"><br/></div><div class="signature">Jens Oehlschlägel<br/><br/><br/></div><div class="signature">> library(ff)<br/>> library(snowfall)<br/>> ncpus <- 2<br/>> n <- 1e8<br/>> system.time(<br/>+ x <- ff(vmode="double", length=n, filename="c:/Temp/x.ff")<br/>+ )<br/> User System verstrichen <br/> 0.01 0.00 0.02 <br/>> # check finalizer, with an explicit filename we should have a 'close' finalizer<br/>> finalizer(x)<br/>[1] "close"<br/>> # if not, set it to 'close' inorder to not let slaves delete x on slave shutdown<br/>> finalizer(x) <- "close"<br/>> sfInit(parallel=TRUE, cpus=ncpus, type="SOCK")<br/>R Version: R version 2.15.0 (2012-03-30) <br/><br/>snowfall 1.84 initialized (using snow 0.3-9): parallel execution on 2 CPUs.<br/><br/>> sfLibrary(ff)<br/>Library ff loaded.<br/>Library ff loaded in cluster.<br/><br/>Warnmeldung:<br/>In library(package = "ff", character.only = TRUE, pos = 2, warn.conflicts = TRUE, :<br/> 'keep.source' is deprecated and will be ignored<br/>> sfExport("x") # note: do not export the same ff multiple times<br/>> # explicitely opening avoids a gc problem<br/>> sfClusterEval(open(x, caching="mmeachflush")) # opening with 'mmeachflush' inststead of 'mmnoflush' is a bit slower but prevents OS write storms when the file is larger than RAM<br/>[[1]]<br/>[1] TRUE<br/><br/>[[2]]<br/>[1] TRUE<br/><br/>> system.time(<br/>+ sfLapply( chunk(x, length=ncpus), function(i){<br/>+ x[i] <- runif(sum(i))<br/>+ invisible()<br/>+ })<br/>+ )<br/> User System verstrichen <br/> 0.00 0.00 30.78 <br/>> system.time(<br/>+ s <- sfLapply( chunk(x, length=ncpus), function(i) quantile(x[i], c(0.05, 0.95)) )<br/>+ )<br/> User System verstrichen <br/> 0.00 0.00 4.38 <br/>> # for completeness<br/>> sfClusterEval(close(x)) <br/>[[1]]<br/>[1] TRUE<br/><br/>[[2]]<br/>[1] TRUE<br/><br/>> csummary(s)<br/> 5% 95%<br/>Min. 0.04998 0.95<br/>1st Qu. 0.04999 0.95<br/>Median 0.05001 0.95<br/>Mean 0.05001 0.95<br/>3rd Qu. 0.05002 0.95<br/>Max. 0.05003 0.95<br/>> # stop slaves<br/>> sfStop()<br/><br/>Stopping cluster<br/><br/>> # with the close finalizer we are responsible for deleting the file explicitely (unless we want to keep it)<br/>> delete(x)<br/>[1] TRUE<br/>> # remove r-side metadata<br/>> rm(x)<br/>> # truly free memory<br/>> gc()<br/><br/></div><div class="signature"><br/></div>
<br/>
<div name="quote" style="margin:10px 5px 5px 10px; padding: 10px 0 10px 10px; border-left:2px solid #C3D9E5; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<div style="margin:0 0 10px 0;">
<b>Gesendet:</b> Donnerstag, 03. Mai 2012 um 00:23 Uhr<br/>
<b>Von:</b> "Jonathan Greenberg" <jgrn@illinois.edu><br/>
<b>An:</b> r-help <r-help@r-project.org>, r-sig-hpc@r-project.org<br/>
<b>Betreff:</b> [R-sig-hpc] Quickest way to make a large "empty" file on disk?
</div>
<div name="quoted-content">
R-helpers:<br/>
<br/>
What would be the absolute fastest way to make a large "empty" file (e.g.<br/>
filled with all zeroes) on disk, given a byte size and a given number<br/>
number of empty values. I know I can use writeBin, but the "object" in<br/>
this case may be far too large to store in main memory. I'm asking because<br/>
I'm going to use this file in conjunction with mmap to do parallel writes<br/>
to this file. Say, I want to create a blank file of 10,000 floating point<br/>
numbers.<br/>
<br/>
Thanks!<br/>
<br/>
--j<br/>
<br/>
-- <br/>
Jonathan A. Greenberg, PhD<br/>
Assistant Professor<br/>
Department of Geography and Geographic Information Science<br/>
University of Illinois at Urbana-Champaign<br/>
607 South Mathews Avenue, MC 150<br/>
Urbana, IL 61801<br/>
Phone: 415-763-5476<br/>
AIM: jgrn307, MSN: jgrn307@hotmail.com, Gchat: jgrn307, Skype: jgrn3007<br/>
<a href="http://www.geog.illinois.edu/people/JonathanGreenberg.html" target="_blank">http://www.geog.illinois.edu/people/JonathanGreenberg.html</a><br/>
<br/>
[[alternative HTML version deleted]]<br/>
<br/>
_______________________________________________<br/>
R-sig-hpc mailing list<br/>
R-sig-hpc@r-project.org<br/>
<a href="https://stat.ethz.ch/mailman/listinfo/r-sig-hpc" target="_blank">https://stat.ethz.ch/mailman/listinfo/r-sig-hpc</a><br/>
</div>
</div><div><br/><br/></div></div></body></html>