[BioC] EBImage Error in .Call("lib_readImages", as.character(files), as.integer(colormode),

Sat Apr 9 08:36:20 CEST 2011

Thanks Henrik, Herve for your suggestions.

I will use the allocMatrix() method to detect the overflow: casting and 
multiplying the dimensions using the double type before comparison to 
INT_MAX.

Cheers,

Greg
---
Gregoire Pau
EMBL Research officer
http://www.embl.de/~gpau/

On 09/04/2011 02:42, Hervé Pagès wrote:
> Hi Greg,
>
> Seems like you have an integer overflow at the C level (src/io.c,
> line 276):
>
> int width, height, nplanes;
>
> image = PROTECT(allocVector(REALSXP, width * height * nplanes));
>
> For big values of width and/or height and/or nplanes, the result of
> the product will be a negative integer.
>
> The way they catch integer multiplication overflow in the R base code
> is with something like (found in src/main/array.c):
>
> if (nrow < 0 || ncol < 0)
> error(_("negative extents to matrix"));
> if ((double)nrow * (double)ncol > INT_MAX)
> error(_("allocMatrix: too many elements specified"));
> n = nrow * ncol;
> PROTECT(s = allocVector(mode, n));
>
> The code above is used in the allocMatrix() function. They also use
> the same trick in alloc3DArray():
>
> if (nrow < 0 || ncol < 0 || nface < 0)
> error(_("negative extents to 3D array"));
> if ((double)nrow * (double)ncol * (double)nface > INT_MAX)
> error(_("alloc3Darray: too many elements specified"));
> n = nrow * ncol * nface;
> PROTECT(s = allocVector(mode, n));
>
> Although this works fine in practice, this is not in theory 100%
> portable (it relies on some subtle assumptions about the size of
> the "int" type in C and how INT_MAX would accurately compare to a
> a double).
>
> Another common technique is to use a type of integer that has at
> least twice the size of the "int" type. Note that, according to
> the ANSI C specs, the "long int" type is not guaranteed to satisfy
> this. Not even the "long long int". But it seems that in practice,
> and probably on most (if not all) platforms where R is supported,
> "int" is 32-bit and "long long int" is 64-bit. So the following
> should be reasonably portable:
>
> int width, height, nplanes;
> unsigned long long n;
>
> n = width;
> n *= height;
> if (n > INT_MAX)
> error("readCellomics: image is too big");
> n *= nplanes;
> if (n > INT_MAX)
> error("readCellomics: image is too big")
> image = PROTECT(allocVector(REALSXP, n));
>
> The advantage of this solution over the "double" solution above is
> that the multiplication is performed only once.
>
> Note that INT_MAX is defined in limits.h
>
> A 3rd solution which seems to be cleaner is to just use unsigned
> int. Since "modulo arithmetic" is well defined on unsigned int
> (according to the C99 standard), one can do:
>
> int width, height, nplanes, n;
> unsigned int uwidth, uheight, unplanes, n1, n2;
>
> uwidth = width;
> uheight = height;
> n1 = uwidth * uheight;
> if ((uheight != 0U) && (n1 / uheight != uwidth))
> error("readCellomics: image is too big");
> unplanes = nplanes;
> n2 = n1 * unplanes;
> if ((unplanes != 0U) && (n2 / unplanes != n1))
> error("readCellomics: image is too big");
> n = n2;
> image = PROTECT(allocVector(REALSXP, n));
>
> But why not just use alloc3DArray() or allocMatrix() in your code in
> the first place. They will not only catch the integer overflow problem
> for you, they will also set the dim attributes on 'image'.
>
> Hope this helps,
>
> H.
>
>
> On 11-04-08 12:17 AM, Gregoire Pau wrote:
>> This is right, Henrik.
>>
>> The image is 52703x20688: this is huge and I cannot display it on my
>> computer. This image would take 52703x20688x8 bytes in memory, i.e.
>> about 8 GBytes. I am not sure if R (even the 64-bit version of R) can
>> handle such big objects.
>>
>> Greg
>> ---
>> Gregoire Pau
>> EMBL Research Officer
>> http://www.embl.de/~gpau/
>>
>> On 07/04/11 18:50, Vladimir Morozov wrote:
>>> Henrik,
>>>
>>> I think you make the right guess.
>>>
>>> Anyway the tiff file is
>>> 0500101820.0.tif -
>>> https://docs.google.com/leaf?id=0B2NLCeM3RRN2ODIxYWVmZjItYzYwNS00YjJjLWE2YjQtYjQwZGUzZTJkOTBi&hl=en
>>>
>>>
>>>
>>> Thanks,
>>>
>>> Vlad
>>>
>>>
>>> -----Original Message-----
>>> From: henrik.bengtsson at gmail.com [mailto:henrik.bengtsson at gmail.com]
>>> On Behalf Of Henrik Bengtsson
>>> Sent: Thursday, April 07, 2011 12:46 PM
>>> To: Vladimir Morozov
>>> Cc: Gregoire Pau; bioconductor at stat.math.ethz.ch
>>> Subject: Re: [BioC] EBImage Error in .Call("lib_readImages",
>>> as.character(files), as.integer(colormode),
>>>
>>> A guess: Your image dimension might be too large. In R,
>>> matrices/arrays are stored as vectors with a dimension attribute
>>> meaning the number of elements is limited by prod(dim(x)) for an array
>>> 'x', not max(dim(x)). Since indices are 32-bit integers in R, you may
>>> be hitting the maximum limit:
>>>
>>>> .Machine$integer.max
>>> [1] 2147483647
>>>
>>> # For your small image that works
>>>> dim<- c(26352,10344,3)
>>>> prod(dim)
>>> [1] 817755264
>>>> (prod(dim)<= .Machine$integer.max)
>>> [1] TRUE
>>>
>>> # For the 90/50=1.8 times larger one
>>>> dim<- 1.8*c(26352,10344,3)
>>>> prod(dim)
>>> [1] 4769148700
>>>> (prod(dim)<= .Machine$integer.max)
>>> [1] FALSE
>>>
>>> /Henrik
>>>
>>>
>>> On Thu, Apr 7, 2011 at 9:23 AM, Vladimir Morozov<vmorozov at als.net>
>>> wrote:
>>>> Hi Gregory,
>>>>
>>>> It dosen't seem to be multiple version problem:
>>>>
>>>> [rstats:ImageAnal] ldd
>>>> /usr/local/lib64/R/library/EBImage/libs/EBImage.so|grep -i magic
>>>> libMagickCore.so.3 => /usr/lib64/libMagickCore.so.3
>>>> (0x00007f264092f000)
>>>> libMagickWand.so.3 => /usr/lib64/libMagickWand.so.3
>>>> (0x00007f263e62c000) [rstats:ImageAnal] which convert /usr/bin/convert
>>>> [rstats:ImageAnal] ldd /usr/bin/convert
>>>> linux-vdso.so.1 => (0x00007fff09bff000)
>>>> libMagickCore.so.3 => /usr/lib64/libMagickCore.so.3
>>>> (0x00007fdad3f45000)
>>>> libMagickWand.so.3 => /usr/lib64/libMagickWand.so.3
>>>> (0x00007fdad3c26000)
>>>> ....
>>>>
>>>>
>>>> I can resize the tiff image with ImageMagic:
>>>> convert -channel B -resize '50%'
>>>> /data/projects/ImageAnal/tiff/0500101820.0.tif
>>>> /data/projects/ImageAnal/tiff/0500101820.05.png
>>>> And then get it into R :
>>>>> x<- readImage('/data/projects/ImageAnal/tiff/0500101820.05.png')
>>>>> dim(x)
>>>> [1] 26352 10344 3
>>>>
>>>> However bigger resolution images fail:
>>>> convert -channel B -resize '90%'
>>>> /data/projects/ImageAnal/tiff/0500101820.0.tif
>>>> /data/projects/ImageAnal/tiff/0500101820.09.png
>>>>
>>>>> x<- readImage('/data/projects/ImageAnal/tiff/0500101820.09.png')
>>>> Error in .Call("lib_readImages", as.character(files),
>>>> as.integer(colormode), :
>>>> negative length vectors are not allowed
>>>>
>>>> So it doesn't look as ImageMagic problem. If you still want the tiff
>>>> file, I can post it
>>>>
>>>> Best
>>>> Vlad
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: Gregoire Pau [mailto:gregoire.pau at embl.de]
>>>> Sent: Thursday, April 07, 2011 11:53 AM
>>>> To: Vladimir Morozov
>>>> Cc: 'bioconductor at stat.math.ethz.ch'
>>>> Subject: Re: [BioC] EBImage Error in .Call("lib_readImages",
>>>> as.character(files), as.integer(colormode),
>>>>
>>>> Hello Vladimir,
>>>>
>>>> If ImageMagick is able to read the image, EBImage should be able to
>>>> do so. Do you have several installed versions of ImageMagick ? To
>>>> which ImageMagick version EBImage is linked to (try otool or ldd on
>>>> EBImage.so or EBImage.dll) ?
>>>>
>>>> TIFF is a not a file format but a file container which can contain
>>>> exotic formats: could you please send me your image for inspection ?
>>>>
>>>> Cheers,
>>>>
>>>> Greg
>>>> ---
>>>> Gregoire Pau
>>>> EMBL Research Officer
>>>> http://www.embl.de/~gpau/
>>>>
>>>> On 07/04/11 16:48, Vladimir Morozov wrote:
>>>>> Hi,
>>>>>
>>>>> I get image reading error:
>>>>>
>>>>>> x<- readImage(
>>>>>
>>>>> "/data/projects/ImageAnal/tiff/0500101820.0.tif"
>>>>>
>>>>> )
>>>>>
>>>>> Error in .Call("lib_readImages", as.character(files),
>>>>> as.integer(colormode), :
>>>>>
>>>>> negative length vectors are not allowed
>>>>>
>>>>>> packageVersion('EBImage')
>>>>>
>>>>> [1] '3.6.0'
>>>>>
>>>>>>
>>>>>
>>>>> ImageMagic can work with this image:
>>>>>
>>>>> convert -channel B -resize '70%'
>>>>> /data/projects/ImageAnal/tiff/0500101820.0.tif
>>>>> /data/projects/ImageAnal/tiff/0500101820.07.jpeg&
>>>>>
>>>>>
>>>>>
>>>>> Tthis image is the bigest image from SVS stack:
>>>>>
>>>>> convert /data/projects/ImageAnal/svs/0500101820.svs
>>>>> /data/projects/ImageAnal/tiff/0500101820.%d.tif
>>>>>
>>>>> [rstats:ImageAnal] ll -h /data/projects/ImageAnal/tiff/0500101820.*
>>>>> -rw-r--r-- 1 vmorozov users 24K 2011-04-06 15:18
>>>>> /data/projects/ImageAnal/tiff/0500101820.1.tif
>>>>> -rw-r--r-- 1 vmorozov users 81M 2011-04-06 15:18
>>>>> /data/projects/ImageAnal/tiff/0500101820.0.tif
>>>>> -rw-r--r-- 1 vmorozov users 155K 2011-04-06 15:18
>>>>> /data/projects/ImageAnal/tiff/0500101820.5.tif
>>>>> -rw-r--r-- 1 vmorozov users 49K 2011-04-06 15:18
>>>>> /data/projects/ImageAnal/tiff/0500101820.4.tif
>>>>> -rw-r--r-- 1 vmorozov users 196K 2011-04-06 15:18
>>>>> /data/projects/ImageAnal/tiff/0500101820.3.tif
>>>>> -rw-r--r-- 1 vmorozov users 3.0M 2011-04-06 15:18
>>>>> /data/projects/ImageAnal/tiff/0500101820.2.tif
>>>>> [rstats:ImageAnal] convert -v
>>>>> Version: ImageMagick 6.6.1-0 2010-07-29 Q16
>>>>> http://www.imagemagick.org
>>>>> Copyright: Copyright (C) 1999-2010 ImageMagick Studio LLC
>>>>> Features: OpenMP
>>>>> .....
>>>>>
>>>>>
>>>>>
>>>>> I have the same problem with other SVS stacks, i.e readImage fails to
>>>>> read the most quality images. it works fine on smaller images
>>>>>
>>>>>
>>>>>
>>>>> Any suggestions would be apriciated
>>>>>
>>>>> Thanks
>>>>>
>>>>>
>>>>>
>>>>> Vladimir Morozov
>>>>> Sr. Computational Biologist
>>>>> ALS Therapy Development Institute
>>>>> 215 First Street, Cambridge MA, 02142 www.als.net<http://www.als.net>
>>>>> Help End ALS! Click Here<http://als.net/EndALS.aspx> or Text
>>>>> "EndALS" to 20222 to make a $5 donation towards research.
>>>>>
>>>>>
>>>>>
>>>>> *********************************************************************
>>>>> *
>>>>> ***************** The information contained in this electronic
>>>>> message is ...{{dropped:21}}
>>>>>
>>>>> _______________________________________________
>>>>> Bioconductor mailing list
>>>>> Bioconductor at r-project.org
>>>>> https://stat.ethz.ch/mailman/listinfo/bioconductor
>>>>> Search the archives:
>>>>> http://news.gmane.org/gmane.science.biology.informatics.conductor
>>>>
>>>>
>>>> **********************************************************************
>>>> ***************** The information contained in this electronic message
>>>> is ...{{dropped:18}}
>>>>
>>>> _______________________________________________
>>>> Bioconductor mailing list
>>>> Bioconductor at r-project.org
>>>> https://stat.ethz.ch/mailman/listinfo/bioconductor
>>>> Search the archives:
>>>> http://news.gmane.org/gmane.science.biology.informatics.conductor
>>>>
>>>
>>>
>>> ***************************************************************************************
>>>
>>>
>>> The information contained in this electronic message i...{{dropped:3}}
>>
>> _______________________________________________
>> Bioconductor mailing list
>> Bioconductor at r-project.org
>> https://stat.ethz.ch/mailman/listinfo/bioconductor
>> Search the archives:
>> http://news.gmane.org/gmane.science.biology.informatics.conductor
>
>