[R-SIG-Mac] Memory error

[stefano iacus]

Not specifically in the statistical problem but in the dimensionality  
of your problem.
stefano

On Nov 28, 2005, at 2:27 PM, Alan Olav Bergland wrote:

> The data set is fairly big.
>
> Again, the model I was trying to run was:
> lme(ovn~tlc+geno+log(food), clinal, random=~tlc+geno+log(food)|block/
> lat)
>
> where,
>
> "ovn" is a continuous, normal (length=999)
> "tlc" is continuous, normal, definitely random
> "food" could either be a factor with 6 levels or a numerical variable
> (making it log(food) makes a more even distribution).  Even though it
> is an experimental treatment, I would like to treat it as random
> because I am interested in the slope and its interaction with the
> "geno" variable (although, note, there are no interactions in the
> model above, as putting them in at this point would cause even more
> problems)
> "geno" is genotype, 12 total, 4 from each "lat."  "geno" should
> really be treated as random too, because they are random draws from
> all genotypes in a population
> "lat" is latitude, 3 total.  Right now I'm treating it as fixed.
> "block" is equivalent to replicate, 2 total.
>
> The experimental design was the following:
>
> 12 genotypes, from 3 latitudes, reared under 6 conditions, replicated
> twice with ~15-20 individuals measured from each geno x food x block
> interaction.  Because individuals from each g x f x b were reared in
> the same rearing chamber, they are not independent thus some grouping
> structure is necessary.
>
> I appreciate your curiosity on this problem.  I know, however, that
> this forum is not the place to hack out statistical questions, but
> this memory thing seemed specific to Macs.
>
> Cheers,
> Alan
>
>
>
>
> On Nov 28, 2005, at 8:01 AM, stefano iacus wrote:
>
>> I don't think this is necessarily a user problem of allocating big
>> chunks of memory, or at least I did experienced last days the same
>> kind of issue by iterating some linear optimization problem of very
>> small dimension (something like 5 rows x 5 columns matrix).
>> In my particular problem it was a sequence on independent calls to
>> lpSolve on very small problems. Looking at "top" command on the
>> shell, I've seen a growing number of vmem (VSIZE?) almost linearly
>> with iteration up to 3.5 giga.
>> This is also occurring after forcing gc(). If i stop the iteration
>> before the alloc failure, the previously allocated ram never gets
>> back until I quit R itself.
>> I'm not able to debug myself, but I can try to provide a
>> reproducible example (not these days, but later on)
>> I've faced the same problem on a G4 and a dual G5.
>>
>> I'm curious to see the dimension of the data Alan is using, but I'm
>> confident that this is not where the problem lies.
>>
>> stefano
>>
>>
>> On Nov 28, 2005, at 1:31 AM, Simon Urbanek wrote:
>>
>>>
>>> On Nov 27, 2005, at 4:37 PM, Alan Olav Bergland wrote:
>>>
>>>> When I attempt to run a rather hefty lme model, I get the following
>>>> error message:
>>>>
>>>>> clinal7.lme<-lme(ovn~tlc+geno+log(food), clinal, random=~tlc+geno
>>>> +log(food)|block/lat)
>>>> Error in logLik.lmeStructInt(lmeSt, lmePars) :
>>>> 	Calloc could not allocate (500237956 of 8) memory
>>>> R(1049,0xa000ed68) malloc: *** vm_allocate(size=4001906688) failed
>>>> (error code=3)
>>>> R(1049,0xa000ed68) malloc: *** error: can't allocate region
>>>> R(1049,0xa000ed68) malloc: *** set a breakpoint in szone_error to
>>>> debug
>>>>
>>>>
>>>> I'm running R Version 2.2.0  (2005-10-06 r35749) on a iMac running
>>>> OS  10.4.3
>>>>
>>>>
>>>> Any suggestions?
>>>
>>> Reformulate your problem.
>>> The lme you run with that data attempts to allocate 4GB of memory
>>> which is a bit too much. Even if you put 8GB in a G5 and run 64-bit
>>> version of R it's likely to run out of memory or to take forever ...
>>> Maybe someone will be able to help you to solve your problem in a
>>> different way if you specify more precisely what you are trying  
>>> to do
>>> (including size of the data etc.).
>>>
>>> Cheers,
>>> Simon
>>>
>>> _______________________________________________
>>> R-SIG-Mac mailing list
>>> R-SIG-Mac at stat.math.ethz.ch
>>> https://stat.ethz.ch/mailman/listinfo/r-sig-mac
>>>
>>
>
> _______________________________________________
> R-SIG-Mac mailing list
> R-SIG-Mac at stat.math.ethz.ch
> https://stat.ethz.ch/mailman/listinfo/r-sig-mac