[R] q about memory usage progression

[Juliet Hannah]

One more note. In case it is helpful, I am including the code for my loop:

# data is read in

numSNPs <- ncol(myData);
pvalues <- rep(-1,numSNPs);
names(pvalues) <- colnames(myData);


for (SNPnum in 1:numSNPs)
{
   is.na(pvalues[SNPnum]) <- TRUE;
   try({
      fit.yags <- yags(log(myPhenos$PHENOTYPE) ~
myPhenos$AGE+myPhenos$SEX*myData[,SNPnum], id=myPhenos$id,
family=gaussian,corstr="exchangeable",alphainit=0.05)
      z.gee <- fit.yags at coefficients[5]/sqrt(fit.yags at robust.parmvar[5,5]);
      pval <- 2 * pnorm(abs(z.gee), lower.tail = FALSE);
      pvalues[SNPnum] <- pval;
})
}

pvalues <- format(pvalues,digits=3);

On Mon, Dec 29, 2008 at 11:59 AM, Juliet Hannah <juliet.hannah at gmail.com> wrote:
> I monitored the usage of memory on a script that I ran. It ran 30K
> regressions and it stores p-values for one of the
> coefficients. It read in a file that has 3000 rows and about 30K
> columns. The size of the file is about 170 MB.
>
> My understanding is that memory usage started out at 2.2G and went up to 23G:
>
>
> cpu=00:03:08, mem=172.75822 GBs, io=0.00000, vmem=2.224G, maxvmem=2.224G
> cpu=00:42:35, mem=29517.64894 GBs, io=0.00000, vmem=23.612G, maxvmem=23.612G
>
> I know very little about how memory works, but I thought the hardest
> part would be reading the file in. Could
> someone explain why there is such a substantial increase over the
> course of the script.
>
> Thanks,
>
> Juliet
>