[Rd] R CMD check --force-multiarch does not install all the archs for testing

[Hervé Pagès]

Hi Uwe,

On 11-06-29 10:50 AM, Uwe Ligges wrote:
>
>
> On 28.06.2011 23:11, Hervé Pagès wrote:
>> Simon,
>>
>> On 11-06-28 01:44 PM, Simon Urbanek wrote:
>>>
>>> On Jun 28, 2011, at 3:45 PM, Hervé Pagès wrote:
>>>
>>>> Hi Simon,
>>>>
>>>> On 11-06-28 12:19 PM, Simon Urbanek wrote:
>>>>>
>>>>> On Jun 28, 2011, at 3:01 PM, Hervé Pagès wrote:
>>>>>
>>>>>> Hi Uwe,
>>>>>>
>>>>>> On 11-06-28 01:44 AM, Uwe Ligges wrote:
>>>>>>>
>>>>>>>
>>>>>>> On 28.06.2011 01:31, Hervé Pagès wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> Why isn't 'R CMD check --force-multiarch' installing the package
>>>>>>>> for all the architectures that are going to be checked?
>>>>>>>
>>>>>>> Hervé,
>>>>>>>
>>>>>>> no, since it cannot know that that you need
>>>>>>>
>>>>>>> --merge-multiarch
>>>>>>>
>>>>>>> as an additional install flag for this particular package.
>>>>>>
>>>>>> Why not just use this flag anyway? Does it hurt to use it on
>>>>>> packages that don't strictly need it?
>>>>>>
>>>>>
>>>>> It does for two reasons: a) everything is built twice
>>>>
>>>> That's exactly what I want when I do 'R CMD check --force-multiarch'
>>>>
>>>
>>> No, that's not what it does (and I assume you mean --force-biarch). It
>>> builds the package just once, you're simply overriding the default
>>> behavior of checking for configure.win, that's all. The two flags are
>>> orthogonal, --force-biarch makes no sense with --merge-multiarch, they
>>> are for all practical purposes mutually exclusive by definition of
>>> what they do.
>>
>> I really mean --force-multiarch, not --force-biarch. AFAIK 'R CMD check'
>> has no --force-biarch option.
>>
>>>
>>>
>>>>> and b) package authors don't expect the necessity to support
>>>>> --libs-only if the package doesn't require separate build runs.
>>>>
>>>> When specifying --force-multiarch, the user really expects the
>>>> package to be installed for all sub-archs.
>>>>
>>>
>>> ... with the assumption that the package supports it even though it
>>> has a configure script which may to may not work unlike
>>> --merge-multiarch which will always work.
>>
>> Just to clarify:
>>
>> --force-multiarch is an 'R CMD check' option, not an 'R CMD INSTALL'
>> option
>>
>> --merge-multiarch is an 'R CMD INSTALL' option, not an 'R CMD check'
>> option
>>
>> Now you are saying that --merge-multiarch will always work (on Windows
>> of course, all this discussion is about how to achieve multiarch check
>> on Windows). Great, this is what I've been observing too so far!
>
>
> It does not "always work", unfortunately. We need force-biarch on CRAN for:
>
> RGtk2
> rgdal
> rphast
> SQLiteMap
>
>
>
>> On packages with or without native codes, with or without configure.win
>> scripts, etc... It always seems to work. So, again, why isn't
>> 'R CMD check --force-multiarch' installing with --merge-multiarch?
>> Note that I'm not attached to that solution in particular, just trying
>> to suggest an easy fix for 'R CMD check --force-multiarch' (which right
>> now is broken on some packages).
>
> Well, merge-multiarch will result in small overhead. It will try to
> compile the code in ./src for the other architecture and merge that into
> the same package. For packages without any compiled code, it does not
> make sense to ask R to do that since the packages are identical under
> 32-bit and 64-bit, hence you don't want the overhead.

If the package doesn't contain any native code, then trying to compile
the native code for the other arch shouldn't result in a big overhead.
What's the overhead of an if statement like:

   ## add DLL for x64
   if (file.exists("installation_folder/pkgname/libs") &&
       !file.exists("installation_folder/pkgname/libs/x64")) {
       ...
       ## compile and install DLL for x64
       ...
   }
   ## test if installed package can be loaded for x64
   ...

Note that the "add DLL for x64" step is currently doing something
*very* useful, which is to make sure that the package can be loaded for
this arch:

D:\biocbld\bbs-2.9-bioc\meat>..\R\bin\R CMD INSTALL --merge-multiarch 
BSgenome_1.21.3.tar.gz

install for i386

* installing to library 'D:/biocbld/bbs-2.9-bioc/R/library'
* installing *source* package 'BSgenome' ...
** R
** inst
** preparing package for lazy loading

Attaching package: 'IRanges'

The following object(s) are masked from 'package:base':

     Map, cbind, eval, intersect, mapply, order, paste, pmax, pmax.int,
     pmin, pmin.int, rbind, rep.int, setdiff, table, union

** help
*** installing help indices
** building package indices ...
*** tangling vignette sources ...
    'BSgenomeForge.Rnw'
    'GenomeSearching.Rnw'
** testing if installed package can be loaded

add DLL for x64

* installing to library 'D:/biocbld/bbs-2.9-bioc/R/library'
* installing *source* package 'BSgenome' ...
** testing if installed package can be loaded

* DONE (BSgenome)

It's useful even for packages without compiled code since they can
depend on packages that do have compiled code but have been installed
only for 1 arch.

Unfortunately this is something that --force-biarch doesn't do
(it returns success even if compilation for x64 failed).

> For packages that
> include compiled code, you still do not want it, since R does it
> automatically - unless there is a configure.win file.

I want to be able to use the same command for any package.
'R CMD check' looses *a lot* of value if people need to know what's
in a package in order to choose the right command for doing a
multiarch check. I'm not talking about special configuration options
that some packages with compiled code need. I'm talking about the
situation where everything is setup properly so the 2 separate
checks work:

   R --arch i386 CMD check pkg_x.y.z.tar.gz
   R --arch x64 CMD check pkg_x.y.z.tar.gz

but 'R CMD check --force-multi-arch pkg_x.y.z.tar.gz' doesn't because
it fails to install for x64.

So if installing with --merge-multiarch is "almost" the right thing
to do (even though it works for only 100% of the BioC packages and
99.8% of the CRAN packages -- note that this percentage would drop
without using --merge-multiarch), then I do want to use it because
this is the closest I can get to *the* command that works for any
package.

Another reason I want to use it for any package that includes
compiled code, even for packages with a configure.win file, is
because "R automatic bi-arch installation" feature will report
success even if compilation for x64 failed.

>
> Therefore, the option is just an add on that is necessary for only 18
> out of roughly 3000 CRAN packages.
>
> So we have 4 packages that require --force-biarch and 18 requiring
> --merge-multiarch on CRAN. All the others build automatically for both
> architectures if they need to. So no reason to do that all the time.

For our BioC builds, we are lucky that --merge-multiarch works for all
our software packages. Unfortunately we cannot just use

   R CMD check --install-args="--merge-multiarch"

because 'R CMD check' calls 'R CMD INSTALL --merge-multiarch' on the
*extracted* tarball but --merge-multiarch only works on a *source*
tarball. So we use this long and ugly command (that is basically the
2-step command that you provided earlier in this thread, thanks!):

 
http://bioconductor.org/checkResults/2.9/bioc-LATEST/Biobase/moscato1-checksrc.html

Anyway I still think anybody should be able to run a multiarch check.
Not only the CRAN, or R-forge, or BioC build systems. It should be
easy (1 step command). It should not need any "add-on".

Thanks,
H.

>
> Best wishes,
> Uwe
>
>
>
>>> --force-biarch is just a way to flag packages that have configure.win
>>> that has no effect on the binary settings (flags etc.). It forces R to
>>> try multi-arch build in one flight, but it may or may not work
>>> depending on the package. It is a way to save time by not running
>>> --merge-multiarch (and thus building the package twice).
>>
>> --force-biarch is an 'R CMD INSTALL' option that I don't use. Why would
>> I use something that might fail when I can use --merge-multiarch which
>> always works.
>  >
>> Thanks,
>> H.
>>
>>>
>>> Cheers,
>>> Simon
>>>
>>>
>>>>>
>>>>> The cross-platform way is to not use --merge-multiarch but use
>>>>> --libs-only instead as needed (easy to check after the first arch
>>>>> run which will tell you whether it's needed or not). I suspect that
>>>>> --merge-multiarch is just a convenience shortcut (and it's unclear
>>>>> to me why it's Windows-only...).
>>>>
>>>> A great convenience indeed as it allows to do the multiarch install in
>>>> a single step. And it's unclear to me too why it's Windows-only but I
>>>> would have hoped you would know...
>>>>
>>>> Thanks,
>>>> H.
>>>>
>>>>
>>>>>
>>>>> Cheers,
>>>>> Simon
>>>>>
>>>>>
>>>>>>> You will have to check it in repository maintainer's mode (as the
>>>>>>> CRAN
>>>>>>> maintainers do everywhere). Essentially this is for me (when also
>>>>>>> producing WIndows binaries):
>>>>>>>
>>>>>>>
>>>>>>> Step 1: Installation
>>>>>>>
>>>>>>> R CMD INSTALL --pkglock --compact-docs --build --merge-multiarch
>>>>>>> --library="D:/path/to/library" fabia_1.5.0.tar.gz>
>>>>>>> fabia-install.out 2>&1
>>>>>>>
>>>>>>> (where the merge-multiarch part applies only to this package, of
>>>>>>> course)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Step 2: Check (without installation, since that happened before
>>>>>>> already,
>>>>>>> using the install log from step 1)
>>>>>>>
>>>>>>> R CMD check --library="D:/path/to/library" --force-multiarch
>>>>>>> --install="check:fabia-install.out" fabia
>>>>>>
>>>>>> Whaoooo! Would be nice if there was a plan to make 'R CMD check' also
>>>>>> usable by normal people (including the package developer), not just
>>>>>> by a few privileged people that know about those undocumented tricks.
>>>>>>
>>>>>> Thanks,
>>>>>> H.
>>>>>>
>>>>>>>
>>>>>>> Best wishes,
>>>>>>> Uwe
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> For some packages, it only installs for the default arch ('i386').
>>>>>>>> Then testing the package for 'x64' fails.
>>>>>>>>
>>>>>>>> For example,
>>>>>>>>
>>>>>>>> Output of R CMD check --force-multiarch fabia_1.5.0.tar.gz:
>>>>>>>> -----------------------------------------------------------
>>>>>>>> * using log directory 'D:/biocbld/bbs-2.9-bioc/meat/fabia.Rcheck'
>>>>>>>> * using R version 2.14.0 Under development (unstable) (2011-05-30
>>>>>>>> r56020)
>>>>>>>> * using platform: i386-pc-mingw32 (32-bit)
>>>>>>>> * using session charset: ISO8859-1
>>>>>>>> * using option '--no-vignettes'
>>>>>>>> * checking for file 'fabia/DESCRIPTION' ... OK
>>>>>>>> * this is package 'fabia' version '1.5.0'
>>>>>>>> * checking package name space information ... OK
>>>>>>>> * checking package dependencies ... OK
>>>>>>>> * checking if this is a source package ... OK
>>>>>>>> * checking whether package 'fabia' can be installed ... OK
>>>>>>>> * checking installed package size ... OK
>>>>>>>> * checking package directory ... OK
>>>>>>>> * checking for portable file names ... OK
>>>>>>>> * checking DESCRIPTION meta-information ... OK
>>>>>>>> * checking top-level files ... OK
>>>>>>>> * checking index information ... OK
>>>>>>>> * checking package subdirectories ... OK
>>>>>>>> * checking R files for non-ASCII characters ... OK
>>>>>>>> * checking R files for syntax errors ... OK
>>>>>>>> * loading checks for arch 'i386'
>>>>>>>> ** checking whether the package can be loaded ... OK
>>>>>>>> ** checking whether the package can be loaded with stated
>>>>>>>> dependencies
>>>>>>>> ... OK
>>>>>>>> ** checking whether the package can be unloaded cleanly ... OK
>>>>>>>> ** checking whether the name space can be loaded with stated
>>>>>>>> dependencies ... OK
>>>>>>>> ** checking whether the name space can be unloaded cleanly ... OK
>>>>>>>> * loading checks for arch 'x64'
>>>>>>>> ** checking whether the package can be loaded ...Warning: running
>>>>>>>> command '"D:/biocbld/bbs-2.9-bioc/R/bin/x64/Rterm.exe"
>>>>>>>> R_ENVIRON_USER='no_such_file' --no-site-file --no-init-file
>>>>>>>> --no-save
>>>>>>>> --no-restore --slave -f
>>>>>>>> D:\biocbld\bbs-2.9-bioc\tmpdir\RtmpO65p5H\Rin57456988' had status 1
>>>>>>>> ERROR
>>>>>>>> Error: package 'fabia' is not installed for 'arch=x64'
>>>>>>>> Execution halted
>>>>>>>>
>>>>>>>> It looks like this package has a loading problem: see the
>>>>>>>> messages for
>>>>>>>> details.
>>>>>>>>
>>>>>>>> Content of fabia.Rcheck\00install.out:
>>>>>>>> --------------------------------------
>>>>>>>>
>>>>>>>> * installing *source* package 'fabia' ...
>>>>>>>> Building libRcpp.a in RcppSrc...
>>>>>>>> rm -f Rcpp.o libRcpp.a
>>>>>>>> g++ -c Rcpp.cpp -o Rcpp.o -I"D:/biocbld/BBS-2˜1.9-B/R/include"
>>>>>>>> -I"D:/biocbld/BBS-2˜1.9-B/R/src/include" -Wall -O2
>>>>>>>> ar r libRcpp.a Rcpp.o
>>>>>>>> C:\Rtools213\MinGW\bin\ar.exe: creating libRcpp.a
>>>>>>>> ranlib libRcpp.a
>>>>>>>> rm -f Rcpp.o
>>>>>>>> rm -f Rcpp.o
>>>>>>>> ** libs
>>>>>>>> running src/Makefile.win ...
>>>>>>>> rm -f fabia.o fabia.dll *.a *.o *.so *.dll
>>>>>>>> g++ -c fabiac.cpp -o fabia.o -I../RcppSrc
>>>>>>>> -I"D:/biocbld/BBS-2˜1.9-B/R/include" -Wall -O2
>>>>>>>> g++ -shared -s -static-libgcc fabia.o -L../RcppSrc -lRcpp
>>>>>>>> -L"D:/biocbld/BBS-2˜1.9-B/R/bin/i386" -lR -o fabia.dll
>>>>>>>> rm -f fabia.o *.a *.o *.so
>>>>>>>> installing to
>>>>>>>> D:/biocbld/bbs-2.9-bioc/meat/fabia.Rcheck/fabia/libs/i386
>>>>>>>> ** R
>>>>>>>> ** demo
>>>>>>>> ** inst
>>>>>>>> ** preparing package for lazy loading
>>>>>>>> Creating a generic function for "plot" from package "graphics" in
>>>>>>>> package "fabia"
>>>>>>>> ** help
>>>>>>>> *** installing help indices
>>>>>>>> ** building package indices ...
>>>>>>>> *** tangling vignette sources ...
>>>>>>>> 'fabia.Rnw'
>>>>>>>> ** testing if installed package can be loaded
>>>>>>>>
>>>>>>>> * DONE (fabia)
>>>>>>>>
>>>>>>>> The source tarball for this package is available here:
>>>>>>>> http://bioconductor.org/packages/2.9/bioc/html/fabia.html
>>>>>>>>
>>>>>>>> What command should be used to perform a multiarch check of this
>>>>>>>> package?
>>>>>>>>
>>>>>>>> This is on a 64-bit Windows Server 2008 R2 Enterprise machine
>>>>>>>> using a
>>>>>>>> recent combined Windows 32/64 bit binary of R-devel from CRAN.
>>>>>>>>
>>>>>>>> Thanks!
>>>>>>>> H.
>>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Hervé Pagès
>>>>>>
>>>>>> Program in Computational Biology
>>>>>> Division of Public Health Sciences
>>>>>> Fred Hutchinson Cancer Research Center
>>>>>> 1100 Fairview Ave. N, M1-B514
>>>>>> P.O. Box 19024
>>>>>> Seattle, WA 98109-1024
>>>>>>
>>>>>> E-mail: hpages at fhcrc.org
>>>>>> Phone: (206) 667-5791
>>>>>> Fax: (206) 667-1319
>>>>>>
>>>>>> ______________________________________________
>>>>>> R-devel at r-project.org mailing list
>>>>>> https://stat.ethz.ch/mailman/listinfo/r-devel
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Hervé Pagès
>>>>
>>>> Program in Computational Biology
>>>> Division of Public Health Sciences
>>>> Fred Hutchinson Cancer Research Center
>>>> 1100 Fairview Ave. N, M1-B514
>>>> P.O. Box 19024
>>>> Seattle, WA 98109-1024
>>>>
>>>> E-mail: hpages at fhcrc.org
>>>> Phone: (206) 667-5791
>>>> Fax: (206) 667-1319
>>>>
>>>>
>>>
>>
>>


-- 
Hervé Pagès

Program in Computational Biology
Division of Public Health Sciences
Fred Hutchinson Cancer Research Center
1100 Fairview Ave. N, M1-B514
P.O. Box 19024
Seattle, WA 98109-1024

E-mail: hpages at fhcrc.org
Phone:  (206) 667-5791
Fax:    (206) 667-1319