[BioC] Error: cannot allocate vector of size 649.8 Mb

Fri Aug 20 22:53:25 CEST 2010

On 08/20/2010 12:53 PM, Martin Morgan wrote:
> On 08/20/2010 12:12 PM, Marco Ortiz wrote:
>> Yes, i already checked the permissions
>>
>> $ ls -l s6_plantula.fq
>> -rwxrwxrwx 1 marco marco 2618477626 2010-08-11 18:38 s6_plantula.fq
>>
>> any other idea?
> 
> I doubt it's a memory problem.

I'll revise that, and say that the problem occurs when the file is being
opened, not when it is being read in. It could be that on your 32-bit system

>>>> i486-pc-linux-gnu

_representing_ the file _size_ causes the file open to fail.

I think your best bets will be to either split the file using standard
Linux utilities such as 'split' (though you'll need to make sure your
split is on a fastq record boundary) or to locate a 64bit machine wiith
ample memory.

Martin

> Do both of these work in R
> 
>   readLines("s6_plantula.fq", 10)
>   readLines(gzcon(file("s_6_plantula.fq")), 10)
> 
> The permissions are weird (I would have guessed -rw-r--r-- or something)
> but I doubt this is a problem. I guess this is supposed to be a plain
> text file; is it? e.g.,
> 
>   head s6_plantula.fq
> 
> giving something reasonable. Is it mounted on an unusual file system,
> e.g., can you read another file from the same location? What about just
> a small fraction of this file, e.g.,
> 
>   head -n 1024 s6_plantula.fq > some_file.fq
> 
> Also not likely to be a problem, but updating to a current R / ShortRead.
> 
> Martin
> 
>>
>>
>> On Fri, Aug 20, 2010 at 1:45 PM, Martin Morgan <mtmorgan at fhcrc.org> wrote:
>>
>>> On 08/20/2010 11:22 AM, Marco Ortiz wrote:
>>>> Hi
>>>>
>>>> Thanks to respond. I checked and the file path is correctly because i
>>> could
>>>> open a similar file but about 38 Mb, that's why i  thought that the
>>> memory
>>>> was the problem, here is:
>>>>
>>>>
>>>>
>>>>> control <- readFastq("ControlB_2A8_130_50.250k.fq",
>>>> qualityType="SFastqQuality")
>>>>> seedling <- readFastq("s6_plantula.fq", qualityType="SFastqQuality")
>>>> Error en .local(dirPath, pattern, ...) : cannot open file s6_plantula.fq
>>>
>>> From the shell, do you have read permissions for this file?
>>>
>>>  % ls -l s6_plantula.fq
>>>
>>> Martin
>>>
>>>>> control
>>>> class: ShortReadQ
>>>> length: 250000 reads; width: 36 cycles
>>>>
>>>>
>>>>
>>>>
>>>> Here the sessionInfo(), but i'm not sure that it has to give me the info
>>>> about the hardware andf software
>>>>
>>>>
>>>>
>>>>> sessionInfo()
>>>> R version 2.10.1 (2009-12-14)
>>>> i486-pc-linux-gnu
>>>>
>>>> locale:
>>>>  [1] LC_CTYPE=es_MX.utf8       LC_NUMERIC=C
>>>>  [3] LC_TIME=es_MX.utf8        LC_COLLATE=es_MX.utf8
>>>>  [5] LC_MONETARY=C             LC_MESSAGES=es_MX.utf8
>>>>  [7] LC_PAPER=es_MX.utf8       LC_NAME=C
>>>>  [9] LC_ADDRESS=C              LC_TELEPHONE=C
>>>> [11] LC_MEASUREMENT=es_MX.utf8 LC_IDENTIFICATION=C
>>>>
>>>> attached base packages:
>>>> [1] stats     graphics  grDevices utils     datasets  methods   base
>>>>
>>>> other attached packages:
>>>> [1] ShortRead_1.4.0    lattice_0.18-3     BSgenome_1.14.2
>>>>  Biostrings_2.14.12
>>>> [5] IRanges_1.4.16
>>>>
>>>> loaded via a namespace (and not attached):
>>>> [1] Biobase_2.6.1 grid_2.10.1   hwriter_1.2
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Fri, Aug 20, 2010 at 11:23 AM, Martin Morgan <mtmorgan at fhcrc.org>
>>> wrote:
>>>>
>>>>> On 08/20/2010 08:19 AM, Marco Ortiz wrote:
>>>>>> * Hi All, ** ** I am new to the world of R and Bioconductor and I had
>>>>>> the** following error when I try to read a fastaq file of 2 Gb****
>>>>>> First I try to run it in Linux, but there i had an error that tell me
>>>>>> *
>>>>>>
>>>>>> * <**
>>>>>>
>>>>>> Error: (dirPath, pattern, ...) cannot open file
>>>>>>
>>>>>> *
>>>>>>
>>>>>> * Then i try to run it in Windows and there i had this error*
>>>>>>
>>>>>> *
>>>>>> *
>>>>>>
>>>>>> *
>>>>>> *
>>>>>>
>>>>>> * > library(ShortRead)*
>>>>>>
>>>>>> *** > plantula <- readFastq("s6_plantula.fq",
>>>>> qualityType="SFastqQuality")*
>>>>>>
>>>>>> *** Error: cannot allocate vector of size 649.8 Mb*
>>>>>
>>>>> This is likely limited memory problem; R is saying 'I allocated some
>>>>> memory, and now I'm trying to allocate 649.8Mb more, and that additional
>>>>> memory allocation fails'. Memory limits on 32-bit windows systems are
>>>>> hard to get around; you're better using a 64-bit Windows or Linux
>>>>> system. Your Linux problem seems really  straight-forward -- you haven't
>>>>> specified the file path correctly. Your R likely has command completion,
>>> so
>>>>>
>>>>>  readFastq("~/
>>>>>
>>>>> and then use the tab key to complete the path.
>>>>>
>>>>> Please include sessionInfo() when you're posting to the list; it makes
>>>>> clear what software and hardware you're using.
>>>>>
>>>>> Martin
>>>>>>
>>>>>> *
>>>>>> *
>>>>>>
>>>>>> * *
>>>>>>
>>>>>> * Then i **set memory.size() at 3000 as well as memory.limit() but i'm
>>>>>> already having the problem*
>>>>>>
>>>>>> * i has a processor of 32 bits, a virtual memory of 3754 Mb*
>>>>>>
>>>>>> *  ** Can someone tell me if this is due to an improper configuration,
>>>>>> lack** of sufficient memory, etc?*
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Martin Morgan
>>>>> Computational Biology / Fred Hutchinson Cancer Research Center
>>>>> 1100 Fairview Ave. N.
>>>>> PO Box 19024 Seattle, WA 98109
>>>>>
>>>>> Location: Arnold Building M1 B861
>>>>> Phone: (206) 667-2793
>>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Martin Morgan
>>> Computational Biology / Fred Hutchinson Cancer Research Center
>>> 1100 Fairview Ave. N.
>>> PO Box 19024 Seattle, WA 98109
>>>
>>> Location: Arnold Building M1 B861
>>> Phone: (206) 667-2793
>>>
>>
>>
>>
> 
> 

-- 
Martin Morgan
Computational Biology / Fred Hutchinson Cancer Research Center
1100 Fairview Ave. N.
PO Box 19024 Seattle, WA 98109

Location: Arnold Building M1 B861
Phone: (206) 667-2793