[Bioc-devel] Mac build (xcms)
Patrick Aboyoun
paboyoun at fhcrc.org
Tue Feb 23 01:01:26 CET 2010
Colin,
Is this a concern or has the xcms dev team had problems with the xcms
binary packages distributed from bioconductor.org?
The build reports can be overwhelming and the relevant reports we are
interested in for this discussion is the "BUILD BIN" step:
http://bioconductor.org/checkResults/2.5/bioc-LATEST/xcms/pitt-buildbin.html
http://bioconductor.org/checkResults/2.6/bioc-LATEST/xcms/pitt-buildbin.html
http://bioconductor.org/checkResults/2.5/bioc-LATEST/xcms/pelham-buildbin.html
http://bioconductor.org/checkResults/2.6/bioc-LATEST/xcms/pelham-buildbin.html
The "Command output" section shows the builds on each of the
architectures linking just fine. This is in stark contrast to the ld
warning messages you get during the "BUILD" (SOURCE) step:
http://bioconductor.org/checkResults/2.5/bioc-LATEST/xcms/pitt-buildsrc.html
http://bioconductor.org/checkResults/2.6/bioc-LATEST/xcms/pitt-buildsrc.html
http://bioconductor.org/checkResults/2.5/bioc-LATEST/xcms/pelham-buildsrc.html
http://bioconductor.org/checkResults/2.6/bioc-LATEST/xcms/pelham-buildsrc.html
For the multi-architecture builds, the messages you want to monitor are
in the former "BUILD BIN" step. These "BUILD BIN" builds use the
--preclean flag that performs the cleanup operation you mentioned in
your e-mail. Does this make sense?
Patrick
On 2/22/10 3:26 PM, Colin A. Smith wrote:
> I think the problem is that the old i386 object files in xcms/src/obiwarp are not getting removed prior to the ppc build. Looking at cleanup_pkg() in /Library/Frameworks/R.framework/Resources/bin/build, it appears that there are two ways for the old object files to be removed:
>
> 1) If a Makevars file is present, it attempts to call make clean. xcms has a "clean:" action in it's Makevars file that looks like it should be removing all object files.
>
> 2) After that, there's a command that appears to remove all *.o, *.s, etc files at the root level of the src directory. I'm not enough of a Perl hacker to know for sure.
>
> My guess, based on the build log messages, is that #1 isn't happening for whatever reason, and #2 is. Adding a test "echo" command to the "clean:" action might help show whether #1 is happening. If it isn't, then the trick would be figuring out how to make it happen.
>
> -Colin
>
> On Feb 22, 2010, at 10:59, Patrick Aboyoun wrote:
>
>
>> Steffen,
>> I am unable to reproduce the problem. Currently the builds are fine and when I run R-2.10/R-devel --arch=ppc, download xcms via biocLite, and run the RUnit tests everything seems to be in order. Perhaps there is some confusion over the warning messages you are receiving during the build process? The BioC build process occurs in three stages for Mac OS X 10.4 and 10.5.
>>
>> 1) Build package source tarball, which is *not* distributed at bioconductor.org. At this step we only test arch=i386 and so the ld warning messages for arch=ppc are not meaningful.
>>
>> 2) Check package source tarball that was built in first step. Again we only test arch=i386.
>>
>> 3) Build binary package tarball. At this step, the build for each architecture (i386, pcc for Mac OS X 10.4 and i386, ppc, x86_64 for Mac OS X 10.5) is performed separately using the (parameterized) command
>>
>> R_ARCH=/{i386/ppc} R CMD INSTALL --preclean --library=xcms.buildbin-libdir --libs-only xcms_VERSION.tar.gz
>>
>> The resulting shared objects from each of the architecture builds are then bundled together in the *.tgz Mac OS X binary package tarball. It is this multi-architecture collection that is distributed at bioconductor.org.
>>
>> If I am misunderstanding your concerns, let me know.
>>
>> Cheers,
>> Patrick
>>
>>
>> On 2/20/10 11:52 AM, Steffen Neumann wrote:
>>
>>> Hi,
>>>
>>> XCMS has link problems on MacOS ppc build, see Ralf's mail below.
>>>
>>> We had a similar issue april last year
>>> http://groups.google.de/group/xcms-devel/browse_thread/thread/bb2584a7c015e56e/a24d1436bfc5d552?hl=de&ie=UTF-8&oe=utf-8&q=Re:+xcms+and+parallel+make#a24d1436bfc5d552
>>> which got fixed by Patrick Aboyoun back then. Patrick's answer
>>> back then is at the end of this mail.
>>>
>>> Since R-2.10 or so R CMD build moved over to some perl code,
>>> which might have re-introduced this similar problem unnoticed.
>>> Could that be ?
>>>
>>> If we'd add a unit test to XCMS (which are disabled
>>> in the BioC builds ?!) or added that obiwarp() call to a vignette,
>>> we'd have a broken ("ERROR") build on MacOS. This way
>>> only the users will notice, which is even worse.
>>>
>>> Any clues ?
>>>
>>> Yours,
>>> Steffen
>>>
>>> On Fri, 2010-02-19 at 16:10 -0800, Ralf Tautenhahn wrote:
>>>
>>>
>>>> I am not a Mac person, and I am not familiar with these multiarch
>>>> builds that Mac seems to use.
>>>>
>>>> It seems the current builds of release and devel for Mac
>>>> http://bioconductor.org/checkResults/2.5/bioc-LATEST/xcms/pelham-buildsrc.html
>>>> http://bioconductor.org/checkResults/2.6/bioc-LATEST/xcms/pelham-buildsrc.html
>>>> are not compiled/linked correctly.
>>>>
>>>> g++-4.0 -arch i386 -I/Library/Frameworks/R.framework/Versions/2.10/Resources/include -I/Library/Frameworks/R.framework/Versions/2.10/Resources/include/i386 -I/usr/local/include -I/sw/include -D_LARGEFILE_SOURCE -I/usr/local/include -fPIC -g -O2 -Wall -fasm-blocks -c obiwarp/mat.cpp -o obiwarp/mat.o
>>>> g++-4.0 -arch i386 -I/Library/Frameworks/R.framework/Versions/2.10/Resources/include -I/Library/Frameworks/R.framework/Versions/2.10/Resources/include/i386 -I/usr/local/include -I/sw/include -D_LARGEFILE_SOURCE -I/usr/local/include -fPIC -g -O2 -Wall -fasm-blocks -c obiwarp/vec.cpp -o obiwarp/vec.o
>>>> g++-4.0 -arch i386 -I/Library/Frameworks/R.framework/Versions/2.10/Resources/include -I/Library/Frameworks/R.framework/Versions/2.10/Resources/include/i386 -I/usr/local/include -I/sw/include -D_LARGEFILE_SOURCE -I/usr/local/include -fPIC -g -O2 -Wall -fasm-blocks -c obiwarp/xcms_dynprog.cpp -o obiwarp/xcms_dynprog.o
>>>> g++-4.0 -arch i386 -I/Library/Frameworks/R.framework/Versions/2.10/Resources/include -I/Library/Frameworks/R.framework/Versions/2.10/Resources/include/i386 -I/usr/local/include -I/sw/include -D_LARGEFILE_SOURCE -I/usr/local/include -fPIC -g -O2 -Wall -fasm-blocks -c obiwarp/xcms_lmat.cpp -o obiwarp/xcms_lmat.o
>>>> g++-4.0 -arch i386 -I/Library/Frameworks/R.framework/Versions/2.10/Resources/include -I/Library/Frameworks/R.framework/Versions/2.10/Resources/include/i386 -I/usr/local/include -I/sw/include -D_LARGEFILE_SOURCE -I/usr/local/include -fPIC -g -O2 -Wall -fasm-blocks -c obiwarp/xcms_obiwarp.cpp -o obiwarp/xcms_obiwarp.o
>>>> gcc-4.2 -arch i386 -std=gnu99 -I/Library/Frameworks/R.framework/Versions/2.10/Resources/include -I/Library/Frameworks/R.framework/Versions/2.10/Resources/include/i386 -I/usr/local/include -I/sw/include -D_LARGEFILE_SOURCE -I/usr/local/include -fPIC -g -O2 -Wall -c mzClust_hclust.c -o mzClust_hclust.o
>>>> gcc-4.2 -arch i386 -std=gnu99 -I/Library/Frameworks/R.framework/Versions/2.10/Resources/include -I/Library/Frameworks/R.framework/Versions/2.10/Resources/include/i386 -I/usr/local/include -I/sw/include -D_LARGEFILE_SOURCE -I/usr/local/include -fPIC -g -O2 -Wall -c mzROI.c -o mzROI.o
>>>> gcc-4.2 -arch i386 -std=gnu99 -I/Library/Frameworks/R.framework/Versions/2.10/Resources/include -I/Library/Frameworks/R.framework/Versions/2.10/Resources/include/i386 -I/usr/local/include -I/sw/include -D_LARGEFILE_SOURCE -I/usr/local/include -fPIC -g -O2 -Wall -c netCDF.c -o netCDF.o
>>>> g++-4.0 -arch i386 -I/Library/Frameworks/R.framework/Versions/2.10/Resources/include -I/Library/Frameworks/R.framework/Versions/2.10/Resources/include/i386 -I/usr/local/include -I/sw/include -D_LARGEFILE_SOURCE -I/usr/local/include -fPIC -g -O2 -Wall -fasm-blocks -c ramp_base64.cpp -o ramp_base64.o
>>>> g++-4.0 -arch i386 -I/Library/Frameworks/R.framework/Versions/2.10/Resources/include -I/Library/Frameworks/R.framework/Versions/2.10/Resources/include/i386 -I/usr/local/include -I/sw/include -D_LARGEFILE_SOURCE -I/usr/local/include -fPIC -g -O2 -Wall -fasm-blocks -c ramp.cpp -o ramp.o
>>>> ramp.cpp: In function ‘char* rampValidFileType(const char*)’:
>>>> ramp.cpp:1459: warning: ‘result’ may be used uninitialized in this function
>>>> g++-4.0 -arch i386 -I/Library/Frameworks/R.framework/Versions/2.10/Resources/include -I/Library/Frameworks/R.framework/Versions/2.10/Resources/include/i386 -I/usr/local/include -I/sw/include -D_LARGEFILE_SOURCE -I/usr/local/include -fPIC -g -O2 -Wall -fasm-blocks -c rampR.cpp -o rampR.o
>>>> gcc-4.2 -arch i386 -std=gnu99 -I/Library/Frameworks/R.framework/Versions/2.10/Resources/include -I/Library/Frameworks/R.framework/Versions/2.10/Resources/include/i386 -I/usr/local/include -I/sw/include -D_LARGEFILE_SOURCE -I/usr/local/include -fPIC -g -O2 -Wall -c util.c -o util.o
>>>> gcc-4.2 -arch i386 -std=gnu99 -I/Library/Frameworks/R.framework/Versions/2.10/Resources/include -I/Library/Frameworks/R.framework/Versions/2.10/Resources/include/i386 -I/usr/local/include -I/sw/include -D_LARGEFILE_SOURCE -I/usr/local/include -fPIC -g -O2 -Wall -c xcms.c -o xcms.o
>>>> g++-4.2 -arch i386 -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -single_module -multiply_defined suppress -L/usr/local/lib -o xcms.so obiwarp/mat.o obiwarp/vec.o obiwarp/xcms_dynprog.o obiwarp/xcms_lmat.o obiwarp/xcms_obiwarp.o mzClust_hclust.o mzROI.o netCDF.o ramp_base64.o ramp.o rampR.o util.o xcms.o -L/usr/local/lib -L/sw/lib -lnetcdf -lz -F/Library/Frameworks/R.framework/.. -framework R -Wl,-framework -Wl,CoreFoundation
>>>> ** arch - ppc
>>>> gcc-4.2 -arch ppc -std=gnu99 -I/Library/Frameworks/R.framework/Versions/2.10/Resources/include -I/Library/Frameworks/R.framework/Versions/2.10/Resources/include/ppc -I/usr/local/include -I/sw/include -D_LARGEFILE_SOURCE -I/usr/local/include -fPIC -g -O2 -c mzClust_hclust.c -o mzClust_hclust.o
>>>> gcc-4.2 -arch ppc -std=gnu99 -I/Library/Frameworks/R.framework/Versions/2.10/Resources/include -I/Library/Frameworks/R.framework/Versions/2.10/Resources/include/ppc -I/usr/local/include -I/sw/include -D_LARGEFILE_SOURCE -I/usr/local/include -fPIC -g -O2 -c mzROI.c -o mzROI.o
>>>> gcc-4.2 -arch ppc -std=gnu99 -I/Library/Frameworks/R.framework/Versions/2.10/Resources/include -I/Library/Frameworks/R.framework/Versions/2.10/Resources/include/ppc -I/usr/local/include -I/sw/include -D_LARGEFILE_SOURCE -I/usr/local/include -fPIC -g -O2 -c netCDF.c -o netCDF.o
>>>> g++-4.0 -arch ppc -I/Library/Frameworks/R.framework/Versions/2.10/Resources/include -I/Library/Frameworks/R.framework/Versions/2.10/Resources/include/ppc -I/usr/local/include -I/sw/include -D_LARGEFILE_SOURCE -I/usr/local/include -fPIC -g -O2 -c ramp_base64.cpp -o ramp_base64.o
>>>> g++-4.0 -arch ppc -I/Library/Frameworks/R.framework/Versions/2.10/Resources/include -I/Library/Frameworks/R.framework/Versions/2.10/Resources/include/ppc -I/usr/local/include -I/sw/include -D_LARGEFILE_SOURCE -I/usr/local/include -fPIC -g -O2 -c ramp.cpp -o ramp.o
>>>> g++-4.0 -arch ppc -I/Library/Frameworks/R.framework/Versions/2.10/Resources/include -I/Library/Frameworks/R.framework/Versions/2.10/Resources/include/ppc -I/usr/local/include -I/sw/include -D_LARGEFILE_SOURCE -I/usr/local/include -fPIC -g -O2 -c rampR.cpp -o rampR.o
>>>> gcc-4.2 -arch ppc -std=gnu99 -I/Library/Frameworks/R.framework/Versions/2.10/Resources/include -I/Library/Frameworks/R.framework/Versions/2.10/Resources/include/ppc -I/usr/local/include -I/sw/include -D_LARGEFILE_SOURCE -I/usr/local/include -fPIC -g -O2 -c util.c -o util.o
>>>> gcc-4.2 -arch ppc -std=gnu99 -I/Library/Frameworks/R.framework/Versions/2.10/Resources/include -I/Library/Frameworks/R.framework/Versions/2.10/Resources/include/ppc -I/usr/local/include -I/sw/include -D_LARGEFILE_SOURCE -I/usr/local/include -fPIC -g -O2 -c xcms.c -o xcms.o
>>>> g++-4.2 -arch ppc -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -single_module -multiply_defined suppress -L/usr/local/lib -o xcms.so obiwarp/mat.o obiwarp/vec.o obiwarp/xcms_dynprog.o obiwarp/xcms_lmat.o obiwarp/xcms_obiwarp.o mzClust_hclust.o mzROI.o netCDF.o ramp_base64.o ramp.o rampR.o util.o xcms.o -L/usr/local/lib -L/sw/lib -lnetcdf -lz -F/Library/Frameworks/R.framework/.. -framework R -Wl,-framework -Wl,CoreFoundation
>>>> ld: warning in obiwarp/mat.o, file is not of required architecture
>>>> ld: warning in obiwarp/vec.o, file is not of required architecture
>>>> ld: warning in obiwarp/xcms_dynprog.o, file is not of required architecture
>>>> ld: warning in obiwarp/xcms_lmat.o, file is not of required architecture
>>>> ld: warning in obiwarp/xcms_obiwarp.o, file is not of required architecture
>>>>
>>>> As you can see, the obiwarp part is build for 32 bit, only.
>>>>
>>>> As a result you get this kind of error message on Mac's when trying to
>>>> use obiwarp:
>>>> Error in .Call("R_set_from_xcms", valscantime1, scantime1, mzval,
>>>> mz, :
>>>> C symbol name "R_set_from_xcms" not in load table
>>>>
>>>> (defined in src/obiwarp/xcms_obiwarp.cpp:extern "C" SEXP
>>>> R_set_from_xcms(SEXP valscantime, SEXP scantime, SEXP mzrange, SEXP
>>>> mz, SEXP intensity)
>>>>
>>>> It builds correctly (for the main architecture, only)
>>>> when the --no-multiarch option is used for R CMD INSTALL.
>>>>
>>>>
>>>> Any Mac build gurus out there ?
>>>>
>>>>
>>> ######################################################################################
>>> # This is Patrick's answer from last year, which I can't find
>>> # in any of the public archives:
>>> #
>>>
>>> On Fri, 2009-04-24 at 09:54 -0700, Patrick Aboyoun wrote:
>>>
>>>
>>>> Steffen,
>>>> Per Brian suggestion I changed the build universal binary script on the
>>>> BioC Mac OS X nodes to use the (variable regulated) command
>>>>
>>>> R_ARCH=/$arch $R_CMD CMD INSTALL --preclean --library="$R_LIBS"
>>>> --libs-only "$srcpkg_filepath"
>>>>
>>>> The addition of the --preclean flag solved the problem. There are other
>>>> BioC packages with similar issues that need to be cleaned out now in a
>>>> similar manner, but it appears that BioC packages with embedded source
>>>> libraries no longer have to concern themselves with multi-architecture
>>>> builds. (Problems will still exist for packages that expect the Mac OS X
>>>> machines to have pre-built multi-architecture shared libraries, but that
>>>> is another issue.)
>>>>
>>>>
>>>> Patrick
>>>>
>>>>
>>>> Steffen Neumann wrote:
>>>>
>>>>
>>>>> Hi,
>>>>>
>>>>> the problem we (Patrick) solved with "all: clean $(SHLIB)"
>>>>> was that we have a subdirectory "xcms/src/obiwarp"
>>>>> which contains *.c files and *.o fies after "make all".
>>>>>
>>>>> BUT because MacOS BioC is building i386/PPC/PPC64 consecutively,
>>>>> and presumably calling something like
>>>>>
>>>>> make clean
>>>>> for ARCH in (i386 ppc ppc64) {
>>>>> make all
>>>>> }
>>>>>
>>>>> and *not*
>>>>>
>>>>> for ARCH in (i386 ppc ppc64) {
>>>>> make clean
>>>>> make all
>>>>> }
>>>>>
>>>>> we had linker warnings in http://bioconductor.org/.../xcms/pelham-buildsrc.html
>>>>> about those leftover *.o in the obiwarp subdirectory:
>>>>>
>>>>> g++-4.2 -arch ppc -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -single_module -multiply_defined suppress -L/usr/local/lib -o xcms.so obiwarp/cmdparser.o obiwarp/mat.o obiwarp/vec.o obiwarp/xcms_dynprog.o obiwarp/xcms_lmat.o obiwarp/xcms_obiwarp.o mzClust_hclust.o mzROI.o netCDF.o ramp_base64.o ramp.o rampR.o util.o xcms.o -L/usr/local/lib -L/sw/lib -lnetcdf -lz -F/Library/Frameworks/R.framework/.. -framework R -Wl,-framework -Wl,CoreFoundation
>>>>>
>>>>>
>>>>>
>>>>>> ld: warning in obiwarp/cmdparser.o, file is not of required architecture
>>>>>> ld: warning in obiwarp/mat.o, file is not of required architecture
>>>>>>
>>>>>>
>>>>>>
>>>>> I am happy to get suggestions for a better solution,
>>>>> which 1) allows to organise C sources in subdirectories
>>>>> and 2) works with multiple Arch builds. Maybe these
>>>>> multiarch builds need to call make clean explicitely inbetween ?
>>>>>
>>>>> Yours,
>>>>> Steffen
>>>>>
More information about the Bioc-devel
mailing list