[Rd] Problem with R >3.0.0
Shelton, Samuel
SheltonS at stemcell.ucsf.edu
Tue Aug 20 19:42:42 CEST 2013
Hi all,
Thanks for getting back to me. We would like to move over to v3.0.0 on our
cluster so that we can build matrices larger than 46300*46300 (limit in R
<3.0.0)
but so far we can't get things to work with R v3.0.0 and higher. I am
trying to trouble shoot at the moment and I am now thinking that the
problem is actually with the diag function that has been rewritten in
version 3.0.0.
The problem is definitely with the diag function and it does not occur on
smaller matrices (20000*20000) and I think it maybe a bug.
This illustrates the problem:
This was done on an iMac i5 with OSX 10.8.5 16GB Ram and with R 3.0.1 (but
I do see the same for 3.0.0). This does not occur when I run it with R
2.15.2.
mat1=matrix(rexp(20000^2), 20000)
mat1[1:10,1:10]
[,1] [,2] [,3] [,4] [,5] [,6]
[,7] [,8] [,9] [,10]
[1,] 0.1829090 0.39867734 0.80499126 4.1746377 0.20717066 1.1477365
0.469843567 2.57767543 0.17449595 0.01949358
[2,] 0.5731522 0.15835939 0.29165029 0.6781249 0.64553728 2.4438404
2.140374938 0.40091195 0.51201369 0.98904490
[3,] 0.3250310 0.09934147 0.79962549 0.4933385 0.30473422 0.4556765
0.002640034 0.90606861 2.58772944 0.89884208
[4,] 1.4195017 0.16082660 0.01377838 0.2115803 1.43992672 0.3883675
0.040903805 0.51011305 0.41998024 0.44209926
[5,] 0.8328441 1.10335604 0.11875332 0.1600287 0.17333324 0.3388678
3.206179119 0.52170966 1.03084845 0.05843232
[6,] 1.3179906 0.76376188 1.24231798 0.9424030 0.04440514 1.0237664
2.547528816 1.35629450 0.87983354 0.25236343
[7,] 0.6990544 1.17003075 0.66063936 0.8632534 0.28965611 0.6718020
1.137348735 0.08371053 0.23144290 0.18915132
[8,] 0.9908026 1.20471979 0.08816010 0.2652131 0.03537790 0.3295816
0.144371435 3.03299285 0.09728111 0.39890260
[9,] 0.9557305 0.29196500 0.43955758 0.7332643 2.03457020 0.5858431
2.437192399 0.34689557 0.02039205 0.54898488
[10,] 3.7220703 0.13572389 0.18888673 0.5683698 1.79209016 1.3495723
0.571159401 0.63375850 0.63221987 1.32840290
> mat1[19990:20000,19990:20000]
[,1] [,2] [,3] [,4] [,5] [,6]
[,7] [,8] [,9] [,10] [,11]
[1,] 1.7910910 0.5719982 0.38689588 1.2157545685 0.8530179 1.464105574
0.5986705 1.1623393 0.55244563 0.1770146 0.4326310
[2,] 0.2862914 1.5267870 0.98214645 0.0004617244 0.6395319 0.075217874
0.6725620 0.2403549 0.08436217 0.1435451 0.7487862
[3,] 2.0492301 0.7216115 0.16951284 0.2726676762 2.1893806 1.202518385
0.9897710 1.4813026 2.42517705 0.3398811 0.7285074
[4,] 0.6538994 0.2437594 2.08848881 0.3917212249 0.4441824 0.433749415
1.3022991 1.3695241 0.07057642 0.4296937 2.9307556
[5,] 2.3688094 2.3970048 0.03545232 0.5986997508 0.8914097 0.497023176
0.4210650 1.5337767 0.01141066 1.1562830 1.0572076
[6,] 2.0626934 0.6186995 0.99197835 1.4794321654 0.1549314 1.296227000
0.2790942 0.9327613 0.84131377 0.8782590 0.3279970
[7,] 1.2423823 0.2385994 0.11390071 2.0745023842 1.9152523 0.754186281
1.5474078 2.5899490 5.19298969 1.4680934 1.0537164
[8,] 1.3657070 1.9502828 1.07681438 0.9339731540 1.7532474 0.186193421
1.8699504 1.9187339 5.13248671 0.4621520 0.4753582
[9,] 0.6512000 0.5104660 0.17820166 0.3965162944 0.0919119 0.187808660
0.7391137 0.1574844 0.65985494 0.4066742 0.8072494
[10,] 0.7435028 1.1395666 2.46096009 0.7060164691 1.7965986 0.008278685
0.4642319 0.1582297 1.71676326 0.3662139 0.7864957
[11,] 0.3537041 0.6622001 2.01642141 1.8225423060 0.3295436 1.260737179
0.8430396 0.5132811 0.30547431 1.6088725 0.4001791
diag(mat1)=0
mat1[1:10,1:10]
[,1] [,2] [,3] [,4] [,5] [,6]
[,7] [,8] [,9] [,10]
[1,] 0.0000000 0.39867734 0.80499126 4.1746377 0.20717066 1.1477365
0.469843567 2.57767543 0.17449595 0.01949358
[2,] 0.5731522 0.00000000 0.29165029 0.6781249 0.64553728 2.4438404
2.140374938 0.40091195 0.51201369 0.98904490
[3,] 0.3250310 0.09934147 0.00000000 0.4933385 0.30473422 0.4556765
0.002640034 0.90606861 2.58772944 0.89884208
[4,] 1.4195017 0.16082660 0.01377838 0.0000000 1.43992672 0.3883675
0.040903805 0.51011305 0.41998024 0.44209926
[5,] 0.8328441 1.10335604 0.11875332 0.1600287 0.00000000 0.3388678
3.206179119 0.52170966 1.03084845 0.05843232
[6,] 1.3179906 0.76376188 1.24231798 0.9424030 0.04440514 0.0000000
2.547528816 1.35629450 0.87983354 0.25236343
[7,] 0.6990544 1.17003075 0.66063936 0.8632534 0.28965611 0.6718020
0.000000000 0.08371053 0.23144290 0.18915132
[8,] 0.9908026 1.20471979 0.08816010 0.2652131 0.03537790 0.3295816
0.144371435 0.00000000 0.09728111 0.39890260
[9,] 0.9557305 0.29196500 0.43955758 0.7332643 2.03457020 0.5858431
2.437192399 0.34689557 0.00000000 0.54898488
[10,] 3.7220703 0.13572389 0.18888673 0.5683698 1.79209016 1.3495723
0.571159401 0.63375850 0.63221987 0.00000000
mat1[19990:20000,19990:20000]
[,1] [,2] [,3] [,4] [,5] [,6]
[,7] [,8] [,9] [,10] [,11]
[1,] 0.0000000 0.5719982 0.38689588 1.2157545685 0.8530179 1.464105574
0.5986705 1.1623393 0.55244563 0.1770146 0.4326310
[2,] 0.2862914 0.0000000 0.98214645 0.0004617244 0.6395319 0.075217874
0.6725620 0.2403549 0.08436217 0.1435451 0.7487862
[3,] 2.0492301 0.7216115 0.00000000 0.2726676762 2.1893806 1.202518385
0.9897710 1.4813026 2.42517705 0.3398811 0.7285074
[4,] 0.6538994 0.2437594 2.08848881 0.0000000000 0.4441824 0.433749415
1.3022991 1.3695241 0.07057642 0.4296937 2.9307556
[5,] 2.3688094 2.3970048 0.03545232 0.5986997508 0.0000000 0.497023176
0.4210650 1.5337767 0.01141066 1.1562830 1.0572076
[6,] 2.0626934 0.6186995 0.99197835 1.4794321654 0.1549314 0.000000000
0.2790942 0.9327613 0.84131377 0.8782590 0.3279970
[7,] 1.2423823 0.2385994 0.11390071 2.0745023842 1.9152523 0.754186281
0.0000000 2.5899490 5.19298969 1.4680934 1.0537164
[8,] 1.3657070 1.9502828 1.07681438 0.9339731540 1.7532474 0.186193421
1.8699504 0.0000000 5.13248671 0.4621520 0.4753582
[9,] 0.6512000 0.5104660 0.17820166 0.3965162944 0.0919119 0.187808660
0.7391137 0.1574844 0.00000000 0.4066742 0.8072494
[10,] 0.7435028 1.1395666 2.46096009 0.7060164691 1.7965986 0.008278685
0.4642319 0.1582297 1.71676326 0.0000000 0.7864957
[11,] 0.3537041 0.6622001 2.01642141 1.8225423060 0.3295436 1.260737179
0.8430396 0.5132811 0.30547431 1.6088725 0.0000000
mat1=matrix(rexp(38000^2), 38000)
dim(mat1)
[1] 38000 38000
mat1[1:10,1:10]
[,1] [,2] [,3] [,4] [,5] [,6]
[,7] [,8] [,9] [,10]
[1,] 3.8622815 0.1357886 1.64090976 0.4494637 0.812315613 0.5328906
0.45672475 0.2891504 1.57087882 1.27375802
[2,] 3.8229940 0.1540735 0.30189392 1.2100152 0.003323061 0.7195875
0.60251052 1.9820380 0.18637086 0.05154236
[3,] 2.0411498 0.8371707 0.02714550 1.3032572 0.330472063 0.3502267
0.11908140 0.4155857 3.46471729 0.31890778
[4,] 1.5503390 0.7377494 2.01433675 0.6109255 1.844484309 1.2492693
0.09365743 0.4006219 2.37769616 0.38643521
[5,] 0.4815804 0.5824312 0.61003728 0.4782871 0.526454982 0.1207842
0.93567987 1.7369767 4.47922786 0.20033928
[6,] 0.3791645 0.1015489 1.96832962 1.5417178 1.030250434 0.1362716
1.72807083 0.2570055 0.02127689 0.80225716
[7,] 1.5212795 2.8133952 0.15990367 0.4337506 0.526532536 2.9926685
0.01432572 0.6064162 0.69264596 0.50871566
[8,] 1.2600365 0.1901277 2.34806048 1.1472887 0.141571521 2.0355007
1.12583466 0.3391067 0.18707165 3.71877247
[9,] 3.0197258 2.3693633 0.94571337 0.2756933 0.938999190 1.5892456
0.18612994 1.0498866 1.89162156 1.56643880
[10,] 0.3573243 0.3047595 0.01894034 2.4666841 0.660994174 0.2248711
0.25436398 1.1275389 0.20960212 0.63957112
mat1[37990:38000,37990:38000]
[,1] [,2] [,3] [,4] [,5] [,6]
[,7] [,8] [,9] [,10] [,11]
[1,] 0.8349326 1.32098263 2.68879316 0.52361090 1.4066094 0.00754308
2.4489865 1.76284621 0.09097126 2.1598758 0.8805701
[2,] 2.5754994 1.15753606 1.76895066 3.06645700 0.8767225 0.33247639
1.5726808 0.12698495 0.75271682 6.4336476 0.1330457
[3,] 2.5737957 0.58234929 0.40403205 0.34882433 0.4074048 1.54867135
2.5971068 0.27276140 0.56926494 0.2129180 1.3027215
[4,] 0.9352113 0.46839288 0.41284388 1.40216119 0.8936151 2.98304058
0.4350446 1.14864094 0.26756970 2.6662998 0.1802141
[5,] 1.6430435 1.22137017 0.06943644 0.03251737 0.2142083 0.16964865
1.6099314 0.95709429 0.22884734 0.5087986 0.9048555
[6,] 2.5340159 1.28618183 3.86698388 1.05946189 3.5006776 0.39320471
0.4927357 0.66159842 0.18235678 0.5172896 2.2221550
[7,] 0.1961937 0.17305031 0.62327325 3.14622730 3.4905449 3.08823676
0.5282165 0.48879156 0.11807913 0.7372706 0.9975103
[8,] 0.8392414 0.17107463 1.08732839 0.28981611 0.4722655 0.17587788
1.7426814 0.13985161 1.86885446 1.9031580 1.1088431
[9,] 3.4648342 0.33834943 0.07891645 0.07206860 1.1792628 0.54092203
0.8141844 2.82687085 0.78395229 0.5313417 3.2164664
[10,] 0.1524299 0.02045616 0.67881610 1.51491647 2.4390115 0.89033581
0.9026651 0.46858464 1.47711638 2.1821178 2.2149071
[11,] 0.9397162 0.42376104 0.20034405 1.47672836 1.0461904 0.97296202
2.1658717 0.04487329 0.30611082 1.1680312 0.9952517
diag(mat1)=0
mat1[1:10,1:10]
[,1] [,2] [,3] [,4] [,5] [,6]
[,7] [,8] [,9] [,10]
[1,] 0.0000000 0.1357886 1.64090976 0.4494637 0.812315613 0.5328906
0.45672475 0.2891504 1.57087882 1.27375802
[2,] 3.8229940 0.0000000 0.30189392 1.2100152 0.003323061 0.7195875
0.60251052 1.9820380 0.18637086 0.05154236
[3,] 2.0411498 0.8371707 0.00000000 1.3032572 0.330472063 0.3502267
0.11908140 0.4155857 3.46471729 0.31890778
[4,] 1.5503390 0.7377494 2.01433675 0.0000000 1.844484309 1.2492693
0.09365743 0.4006219 2.37769616 0.38643521
[5,] 0.4815804 0.5824312 0.61003728 0.4782871 0.000000000 0.1207842
0.93567987 1.7369767 4.47922786 0.20033928
[6,] 0.3791645 0.1015489 1.96832962 1.5417178 1.030250434 0.0000000
1.72807083 0.2570055 0.02127689 0.80225716
[7,] 1.5212795 2.8133952 0.15990367 0.4337506 0.526532536 2.9926685
0.00000000 0.6064162 0.69264596 0.50871566
[8,] 1.2600365 0.1901277 2.34806048 1.1472887 0.141571521 2.0355007
1.12583466 0.0000000 0.18707165 3.71877247
[9,] 3.0197258 2.3693633 0.94571337 0.2756933 0.938999190 1.5892456
0.18612994 1.0498866 0.00000000 1.56643880
[10,] 0.3573243 0.3047595 0.01894034 2.4666841 0.660994174 0.2248711
0.25436398 1.1275389 0.20960212 0.00000000
## After calling the diag function the bottom of the matrix is all set to
0.
mat1[37990:38000,37990:38000]
[,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9] [,10] [,11]
[1,] 0 0 0 0 0 0 0 0 0 0 0
[2,] 0 0 0 0 0 0 0 0 0 0 0
[3,] 0 0 0 0 0 0 0 0 0 0 0
[4,] 0 0 0 0 0 0 0 0 0 0 0
[5,] 0 0 0 0 0 0 0 0 0 0 0
[6,] 0 0 0 0 0 0 0 0 0 0 0
[7,] 0 0 0 0 0 0 0 0 0 0 0
[8,] 0 0 0 0 0 0 0 0 0 0 0
[9,] 0 0 0 0 0 0 0 0 0 0 0
[10,] 0 0 0 0 0 0 0 0 0 0 0
[11,] 0 0 0 0 0 0 0 0 0 0 0
It looks like there is an issue with larger matrices when calling diag
function and it has nothing to do with WGCNA.
On 8/20/13 9:43 AM, "Peter Langfelder" <peter.langfelder at gmail.com> wrote:
>Hi Sam,
>
>I assume you mean that correlation for _genes_ (not samples)
>11262:30000 is 0? I am the maintainer of the WGCNA package but
>unfortunately I don't have access to a Mac big enough to try
>30000x30000 correlation matrix, but I would be thankful if you could
>try reproducing the problem with smaller matrices (e.g. 20000x20000)
>and try to produce a small reproducible example by
>generating the data using say rnorm, say like this:
>
>nGenes = 20000 # as small as possible that still produces the error
>nSamples = 100
>datExpr1 = matrix(rnorm(nSamples * nGenes), nSamples, nGenes)
>
>simMat = bicor(datExpr1, use = 'p')
>
>Best,
>
>Peter
>
>On Mon, Aug 19, 2013 at 11:53 AM, Shelton, Samuel
><SheltonS at stemcell.ucsf.edu> wrote:
>> Dear R developer,
>>
>> I am an R user and am currently having a problem with versions of R
>>>3.0.0. We build larger correlation matrices of >30000 with pairwise
>>>correlations made. We have been using a line of code
>>
>> simMat=bicor(datExpr1,use="p")
>>
>> To build the similarity matrix with datExpr1 being a matrix with genes
>>as columns (>30000) and rows being samples. This code works fine on an
>>iMac running OSX 10.8.5 and using R 2.15.2 and the Rblas library. When I
>>try to run the same code with R version 3.0.0 and 3.0.1 it only
>>partially builds the matrix. It will return correlations for samples
>>1:11262 but from 11262:30000 the matrix is full of 0's.
>>
>> We would like to be able to use R 3.0.0 to allow us to build larger
>>correlation matrices on our cluster but we can't use it at the moment
>>due to this problem. My colleague has previously had a similar issue
>>when using snow leopard and this was to do with a compatibility issue
>>between snow leopard and the veclib. Has something changed in version
>>3.0.0 to give an incompatibility with the veclib?
>>
>> Many thanks
>>
>> Sam
>>
>> Samuel Shelton, Ph.D.
>> Postdoctoral Researcher,
>> The Oldham Lab,
>> The institute of Regeneration Medicine,
>> University of California San Francisco,
>> San Francisco, CA 94143
>>
>> email: sheltons at stemcell.ucsf.edu
>>
>>
>> [[alternative HTML version deleted]]
>>
>> ______________________________________________
>> R-devel at r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-devel
More information about the R-devel
mailing list