[R-sig-ME] problems with allocate memory

[Andrew Robinson]

Hi Paul,

I apologize if this has already been discussed and rejected, but I
also echo an earlier suggestion of Douglas's that you should think
about working with a sample of your data instead of all of it.  

Not knowing anything at all about your application :) I'd be very
surprised if you need ~4000000 observations to establish a meaningful
relationship, or to provide a model that develops high-quality
predictions.  I would try to fit the model on a systematic sample of
the data and then ask what would be gained from an increase to the
full dataset.  You might even try a model-averaging approach such as
is used in e.g. imputation; fit the  model ten times to a tenth of the
data and average the results.

Best wishes,

Andrew

On Thu, Dec 22, 2011 at 08:05:28PM +0100, cumuluss at web.de wrote:
> Hi Douglas,
> maybe you mentioned the mail from Andrew. He already tested it but not with the newest CXXR version. I’m not really skilled enough I think, but perhaps I will try to work with CXXR and lme4 to check this out. Probably I will get stranded but on the other hand I’m already stranded.
> I considered my model more carefully but as I wrote before I reduced it a lot without any interaction, no nonlinear term and only six fixed effects. But I was not able to open the results. I don’t see that a more reduced model will tell me something.
> It is clear that I have to think about it again and I hope it leads me to a solution.
> Thank you very much for your effort.
> All the Best
> Paul
> 
> 

-- 
Andrew Robinson  
Deputy Director, ACERA 
Department of Mathematics and Statistics            Tel: +61-3-8344-6410
University of Melbourne, VIC 3010 Australia               (prefer email)
http://www.ms.unimelb.edu.au/~andrewpr              Fax: +61-3-8344-4599
http://www.acera.unimelb.edu.au/

Forest Analytics with R (Springer, 2011) 
http://www.ms.unimelb.edu.au/FAwR/
Introduction to Scientific Programming and Simulation using R (CRC, 2009): 
http://www.ms.unimelb.edu.au/spuRs/