[R-sig-hpc] (help)After writing data in MMF using SEXP structure, can i reference in R?

[Jens Oehlschlägel]

Young-Ju,

> The ff package helped me much.
> I have some question.> After writing data in MMF using SEXP structure, can i reference in R?

ff does not write to MMF using SEXP structure. What is written to file is a vector (or array) of binary data - e.g. doubles - without any header or tail (tail other than completing the last byte in sub-byte vmodes like 'boolean'). 
> If input data is larger than 2GB, Can i reference MMF Data in R?

The high-level functions in ff support up to .Machine$integer.max = 2147483647 elements per atomic ff object. 
For vmode='boolean' this translates to a filesize limit of 0.25GB  
For vmode='double' this translates to filesize limit of 16GB

If your data can be represented in a columnar structure, you can reference larger datasets using ffdf aka ff data.frame: you write each column in a seperate file, reference each column using 'ff' and then bind all columns using 'ffdf'. 

HTH

Jens Oehlschlägel


---------------------
Von: Young-Ju Park  
Gesendet: Jul 28, 2010 4:38:52 AM
An: dadler at uni-goettingen.de,christian_glaeser at gmx.de,onenadi at uni-goettingen.de,Jens.Oehlschlaegel at truecluster.com,wzucchi at uni-goettingen.de
Betreff: (help)After writing data in MMF using SEXP structure, can i reference in R?


Hi all,
 
The ff package helped me much.
 
I have some question.
 
After writing data in MMF using SEXP structure, can i reference in R?
If input data is larger than 2GB,  Can i reference  MMF Data in R?
 
my work environment :
R version : 2.11.1
OS : WinXP Pro sp3

Thanks and best regards.
Park, Young-Ju
 
from Korea.