<html><head></head><body><div style="font-family: Verdana;font-size: 12.0px;"><div class="signature">Jonathan,<br/></div><div class="signature"><br/></div><div class="signature">ff has a utility function file.resize() which allows to give a new filesize in bytes using doubles.<br/></div><div class="signature">See ?file.resize<br/></div><div class="signature"><br/></div><div class="signature">Regards<br/></div><div class="signature"><br/></div><div class="signature">Jens Oehlschlägel<br/></div><div class="signature"><br/></div><div class="signature"><br/></div><div name="quote" style="margin:10px 5px 5px 10px; padding: 10px 0 10px 10px; border-left:2px solid #C3D9E5; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<div style="margin:0 0 10px 0;">
<b>Gesendet:</b> Donnerstag, 27. September 2012 um 21:17 Uhr<br/>
<b>Von:</b> "Jonathan Greenberg" <jgrn@illinois.edu><br/>
<b>An:</b> r-help <r-help@r-project.org>, r-sig-hpc@r-project.org<br/>
<b>Betreff:</b> Re: [R-sig-hpc] Quickest way to make a large "empty" file on disk?
<br/></div>
<div name="quoted-content">
Folks:<br/>
<br/>
Asked this question some time ago, and found what appeared (at first) to be<br/>
the best solution, but I'm now finding a new problem. First off, it seemed<br/>
like ff as Jens suggested worked:<br/>
<br/>
# outdata_ncells = the number of rows * number of columns * number of bands<br/>
in an image:<br/>
out<-ff(vmode="double",length=outdata_ncells,filename=filename)<br/>
finalizer(out) <- close<br/>
close(out)<br/>
<br/>
This was working fine until I attempted to set length to a VERY large<br/>
number: outdata_ncells = 17711913600. This would create a file that is<br/>
131.964GB. Big, but not obscenely so (and certainly not larger than the<br/>
filesystem can handle). However, length appears to be restricted<br/>
by .Machine$integer.max (I'm on a 64-bit windows box):<br/>
> .Machine$integer.max<br/>
[1] 2147483647<br/>
<br/>
Any suggestions on how to solve this problem for much larger file sizes?<br/>
<br/>
--j<br/>
<br/>
<br/>
On Thu, May 3, 2012 at 10:44 AM, Jonathan Greenberg <jgrn@illinois.edu>wrote:<br/>
<br/>
> Thanks, all! I'll try these out. I'm trying to work up something that is<br/>
> platform independent (if possible) for use with mmap. I'll do some tests<br/>
> on these suggestions and see which works best. I'll try to report back in a<br/>
> few days. Cheers!<br/>
><br/>
> --j<br/>
><br/>
><br/>
><br/>
> 2012/5/3 "Jens Oehlschlägel" <jens.oehlschlaegel@truecluster.com><br/>
><br/>
>> Jonathan,<br/>
>><br/>
>> On some filesystems (e.g. NTFS, see below) it is possible to create<br/>
>> 'sparse' memory-mapped files, i.e. reserving the space without the cost of<br/>
>> actually writing initial values.<br/>
>> Package 'ff' does this automatically and also allows to access the file<br/>
>> in parallel. Check the example below and see how big file creation is<br/>
>> immediate.<br/>
>><br/>
>> Jens Oehlschlägel<br/>
>><br/>
>><br/>
>> > library(ff)<br/>
>> > library(snowfall)<br/>
>> > ncpus <- 2<br/>
>> > n <- 1e8<br/>
>> > system.time(<br/>
>> + x <- ff(vmode="double", length=n, filename="c:/Temp/x.ff")<br/>
>> + )<br/>
>> User System verstrichen<br/>
>> 0.01 0.00 0.02<br/>
>> > # check finalizer, with an explicit filename we should have a 'close'<br/>
>> finalizer<br/>
>> > finalizer(x)<br/>
>> [1] "close"<br/>
>> > # if not, set it to 'close' inorder to not let slaves delete x on slave<br/>
>> shutdown<br/>
>> > finalizer(x) <- "close"<br/>
>> > sfInit(parallel=TRUE, cpus=ncpus, type="SOCK")<br/>
>> R Version: R version 2.15.0 (2012-03-30)<br/>
>><br/>
>> snowfall 1.84 initialized (using snow 0.3-9): parallel execution on 2<br/>
>> CPUs.<br/>
>><br/>
>> > sfLibrary(ff)<br/>
>> Library ff loaded.<br/>
>> Library ff loaded in cluster.<br/>
>><br/>
>> Warnmeldung:<br/>
>> In library(package = "ff", character.only = TRUE, pos = 2, warn.conflicts<br/>
>> = TRUE, :<br/>
>> 'keep.source' is deprecated and will be ignored<br/>
>> > sfExport("x") # note: do not export the same ff multiple times<br/>
>> > # explicitely opening avoids a gc problem<br/>
>> > sfClusterEval(open(x, caching="mmeachflush")) # opening with<br/>
>> 'mmeachflush' inststead of 'mmnoflush' is a bit slower but prevents OS<br/>
>> write storms when the file is larger than RAM<br/>
>> [[1]]<br/>
>> [1] TRUE<br/>
>><br/>
>> [[2]]<br/>
>> [1] TRUE<br/>
>><br/>
>> > system.time(<br/>
>> + sfLapply( chunk(x, length=ncpus), function(i){<br/>
>> + x[i] <- runif(sum(i))<br/>
>> + invisible()<br/>
>> + })<br/>
>> + )<br/>
>> User System verstrichen<br/>
>> 0.00 0.00 30.78<br/>
>> > system.time(<br/>
>> + s <- sfLapply( chunk(x, length=ncpus), function(i) quantile(x[i],<br/>
>> c(0.05, 0.95)) )<br/>
>> + )<br/>
>> User System verstrichen<br/>
>> 0.00 0.00 4.38<br/>
>> > # for completeness<br/>
>> > sfClusterEval(close(x))<br/>
>> [[1]]<br/>
>> [1] TRUE<br/>
>><br/>
>> [[2]]<br/>
>> [1] TRUE<br/>
>><br/>
>> > csummary(s)<br/>
>> 5% 95%<br/>
>> Min. 0.04998 0.95<br/>
>> 1st Qu. 0.04999 0.95<br/>
>> Median 0.05001 0.95<br/>
>> Mean 0.05001 0.95<br/>
>> 3rd Qu. 0.05002 0.95<br/>
>> Max. 0.05003 0.95<br/>
>> > # stop slaves<br/>
>> > sfStop()<br/>
>><br/>
>> Stopping cluster<br/>
>><br/>
>> > # with the close finalizer we are responsible for deleting the file<br/>
>> explicitely (unless we want to keep it)<br/>
>> > delete(x)<br/>
>> [1] TRUE<br/>
>> > # remove r-side metadata<br/>
>> > rm(x)<br/>
>> > # truly free memory<br/>
>> > gc()<br/>
>><br/>
>><br/>
>><br/>
>> *Gesendet:* Donnerstag, 03. Mai 2012 um 00:23 Uhr<br/>
>> *Von:* "Jonathan Greenberg" <jgrn@illinois.edu><br/>
>> *An:* r-help <r-help@r-project.org>, r-sig-hpc@r-project.org<br/>
>> *Betreff:* [R-sig-hpc] Quickest way to make a large "empty" file on<br/>
>> disk?<br/>
>> R-helpers:<br/>
>><br/>
>> What would be the absolute fastest way to make a large "empty" file (e.g.<br/>
>> filled with all zeroes) on disk, given a byte size and a given number<br/>
>> number of empty values. I know I can use writeBin, but the "object" in<br/>
>> this case may be far too large to store in main memory. I'm asking because<br/>
>> I'm going to use this file in conjunction with mmap to do parallel writes<br/>
>> to this file. Say, I want to create a blank file of 10,000 floating point<br/>
>> numbers.<br/>
>><br/>
>> Thanks!<br/>
>><br/>
>> --j<br/>
>><br/>
>> --<br/>
>> Jonathan A. Greenberg, PhD<br/>
>> Assistant Professor<br/>
>> Department of Geography and Geographic Information Science<br/>
>> University of Illinois at Urbana-Champaign<br/>
>> 607 South Mathews Avenue, MC 150<br/>
>> Urbana, IL 61801<br/>
>> Phone: 415-763-5476<br/>
>> AIM: jgrn307, MSN: jgrn307@hotmail.com, Gchat: jgrn307, Skype: jgrn3007<br/>
>> <a href="http://www.geog.illinois.edu/people/JonathanGreenberg.html" target="_blank">http://www.geog.illinois.edu/people/JonathanGreenberg.html</a><br/>
>><br/>
>> [[alternative HTML version deleted]]<br/>
>><br/>
>> _______________________________________________<br/>
>> R-sig-hpc mailing list<br/>
>> R-sig-hpc@r-project.org<br/>
>> <a href="https://stat.ethz.ch/mailman/listinfo/r-sig-hpc" target="_blank">https://stat.ethz.ch/mailman/listinfo/r-sig-hpc</a><br/>
>><br/>
>><br/>
>><br/>
><br/>
><br/>
> --<br/>
> Jonathan A. Greenberg, PhD<br/>
> Assistant Professor<br/>
> Department of Geography and Geographic Information Science<br/>
> University of Illinois at Urbana-Champaign<br/>
> 607 South Mathews Avenue, MC 150<br/>
> Urbana, IL 61801<br/>
> Phone: 415-763-5476<br/>
> AIM: jgrn307, MSN: jgrn307@hotmail.com, Gchat: jgrn307, Skype: jgrn3007<br/>
> <a href="http://www.geog.illinois.edu/people/JonathanGreenberg.html" target="_blank">http://www.geog.illinois.edu/people/JonathanGreenberg.html</a><br/>
><br/>
<br/>
<br/>
<br/>
-- <br/>
Jonathan A. Greenberg, PhD<br/>
Assistant Professor<br/>
Department of Geography and Geographic Information Science<br/>
University of Illinois at Urbana-Champaign<br/>
607 South Mathews Avenue, MC 150<br/>
Urbana, IL 61801<br/>
Phone: 217-300-1924<br/>
AIM: jgrn307, MSN: jgrn307@hotmail.com, Gchat: jgrn307, Skype: jgrn3007<br/>
<a href="http://www.geog.illinois.edu/people/JonathanGreenberg.html" target="_blank">http://www.geog.illinois.edu/people/JonathanGreenberg.html</a><br/>
<br/>
[[alternative HTML version deleted]]<br/>
<br/>
_______________________________________________<br/>
R-sig-hpc mailing list<br/>
R-sig-hpc@r-project.org<br/>
<a href="https://stat.ethz.ch/mailman/listinfo/r-sig-hpc" target="_blank">https://stat.ethz.ch/mailman/listinfo/r-sig-hpc</a><br/>
</div>
</div><div><br/><br/></div></div></body></html>