[Rd] cleanup and Makevars

Duncan Temple Lang duncan at wald.ucdavis.edu
Wed Jul 11 21:10:09 CEST 2007 

Hi Kasper

 This is not an immediate solution for your problem,
but a suggestion that the "ideal" solution
might be different from what you are asking for.

In many projects, when we need to compile
the same source file for two or more different
purposes (e.g. architectures, compilation flags, etc.)
we put the .o's from each build into their own sub-directory
or use a different extension for the file.
This way, we don't have to recompile all of the source
files again when we start over.

So rather than removing the .o's just after creating
the .dylib, I think it would make sense for the
mac builds that support multiple architectures
to have slightly different make rules that
use different directories for the .o files.

The more we throw into the installation mechanism,
the more complex it becomes and harder to get
something out later on.

Also, adding facilities to perform
"customized" clean up or arbitrary package
installation customizations would be a lot
easier if the package mechanism were written in R.
It would make a nice project for somebody to do this,
perhaps building one some of the facilities in the inline package
or similar for invoking, e.g. R CMD SHLIB, and so on.


D

Kasper Daniel Hansen wrote:
> On Jul 11, 2007, at 9:30 AM, Prof Brian Ripley wrote:
> 
>> On Mon, 9 Jul 2007, Kasper Daniel Hansen wrote:
>>
>>> Hi
>>>
>>> This is a question prompted by the mac version of R, but as I see it,
>>> it should have broader interest.
>>>
>>> These days the CRAN Mac binary per default compiles every package for
>>> two architectures. First i386 and then ppc. In between the two
>>> compilation runs, any object files in pkgname/src is removed. This
>>> cleanup is necessary since otherwise Make would not recompile the
>>> object files under the next architecture.
>>>
>>> I have a package which includes a large SDK which I have put in
>>>   src/fusion_sdk
>>> (and subdirectories). I take care of that by using a Makevars file
>>> with essentially (the whole Makevars file is reproduced below)
>>>
>>> PKG_SOURCES = \
>>>         fusion_sdk/calvin_files/data/src/CDFData.cpp
>>>
>>> OBJS=$(PKG_SOURCES:.cpp=.o)
>>>
>>> all: $(SHLIB)
>>>
>>> However, this setup does not remove the object files in the
>>> fusion_sdk subdirectory. From a posting on R-sig-mac by Simon Urbanek
>>> I learned that I need to clean up these directories myself.
>>>
>>> I have - per R extensions - attempted to do so using a pkgname/
>>> cleanup script. While this works for cleaning up the sub directories
>>> when I do R CMD build, it seems as if this script is not being run
>>> between compilation runs (is this intentional btw.?).
>> What do you mean by 'compilation runs'?  The 'cleanup' script will  
>> be run by R CMD INSTALL --cleanup (and without --cleanup it is  
>> intentionally not run).
>>
>>> So my question is now: how do I in a simple way clean up my
>>> subdirectories? I would prefer it to be as simple as possible because
>>> so far I have not really needed anything besides a Makevars file. In
>>> fact the impression I have right now from looking at the SHLIB and
>>> INSTALL scripts is that I might need a Makefile that removes the
>>> object files _before_ starting the compilation, as I don't really see
>>> any cleanup process (eg. making a target clean) - but this may very
>>> well be due to my limited understanding of these scripts.
>> I see R CMD INSTALL --clean that runs a clean up afterwards, and  
>> 2.6.0 will also have R CMD INSTALL --preclean in case you forgot -- 
>> clean on the last run.
>>
>>> Any help/hints on how to proceed? Especially if I want it to be
>>> portable?
>> What does R CMD INSTALL --clean not do that you want?
>>
>> If you are talking about R CMD SHLIB, that currently does not clean  
>> up but this has been raised and it will most likely have a --clean  
>> option in 2.6.0 that removes $(OBJECTS).
> 
> Perhaps I was not clear enough: The CRAN binary of R for Mac has two  
> architectures inside $R_HOME/bin/exec, namely "ppc" and "i386". This  
> means that whenever anyone installs a source package I see two  
> compilation runs - the following is a shortened output:
> 
> * Installing to library '/Library/Frameworks/R.framework/Resources/ 
> library'
> * Installing *source* package 'affxparser' ...
> ** libs
> ** arch - i386
> g++-4.0 -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk -no-cpp- 
> precomp -I/Library/Frameworks/R.framework/Resources/include -I/ 
> Library/Frameworks/R.framework/Resources/include/i386 -imacros  
> R_affx_constants.h -Ifusion_sdk/calvin_files/array/src -Ifusion_sdk/ 
> calvin_files/data/src -Ifusion_sdk/calvin_files/exception/src - 
> Ifusion_sdk/calvin_files/fusion/src -Ifusion_sdk/calvin_files/fusion/ 
> src/GCOSAdapter -Ifusion_sdk/calvin_files/fusion/src/CalvinAdapter - 
> Ifusion_sdk/calvin_files/parameter/src -Ifusion_sdk/calvin_files/ 
> parsers/src -Ifusion_sdk/calvin_files/portability/src -Ifusion_sdk/ 
> calvin_files/template/src -Ifusion_sdk/calvin_files/utils/src - 
> Ifusion_sdk/calvin_files/writers/src -Ifusion_sdk/file -Ifusion_sdk/ 
> portability -D_USE_MEM_MAPPING_ -msse3    -fPIC  -g -O2 -Wall -O0 -c  
> fusion_sdk/calvin_files/data/src/CDFData.cpp -o fusion_sdk/ 
> calvin_files/data/src/CDFData.o
> 
> (Now a lot of other files gets compiled)
> 
> g++-4.0 -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk - 
> dynamiclib -Wl,-macosx_version_min -Wl,10.3 -undefined dynamic_lookup  
> -single_module -multiply_defined suppress -L/usr/local/lib -o  
> affxparser.so fusion_sdk/calvin_files/data/src/CDFData.o fusion_sdk/ 
> calvin_files/data/src/CDFProbeGroupInformation.o fusion_sdk/ 
> calvin_files/data/src/CDFProbeInformation.o fusion_sdk/calvin_files/ 
> data/src/CDFProbeSetInformation.o fusion_sdk/calvin_files/data/src/ 
> CDFQCProbeInformation.o fusion_sdk/calvin_files/data/src/ 
> CDFQCProbeSetInformation.o fusion_sdk/calvin_files/data/src/CELData.o  
> fusion_sdk/calvin_files/data/src/DataGroup.o fusion_sdk/calvin_files/ 
> data/src/DataGroupHeader.o fusion_sdk/calvin_files/data/src/DataSet.o  
> fusion_sdk/calvin_files/data/src/DataSetHeader.o fusion_sdk/ 
> calvin_files/data/src/FileHeader.o fusion_sdk/calvin_files/data/src/ 
> GenericData.o fusion_sdk/calvin_files/data/src/GenericDataHeader.o  
> fusion_sdk/calvin_files/exception/src/ExceptionBase.o fusion_sdk/ 
> calvin_files/fusion/src/CalvinAdapter/CalvinCELDataAdapter.o  
> fusion_sdk/calvin_files/fusion/src/FusionBPMAPData.o fusion_sdk/ 
> calvin_files/fusion/src/FusionCDFData.o fusion_sdk/calvin_files/ 
> fusion/src/FusionCDFQCProbeSetNames.o fusion_sdk/calvin_files/fusion/ 
> src/FusionCELData.o fusion_sdk/calvin_files/fusion/src/GCOSAdapter/ 
> GCOSCELDataAdapter.o fusion_sdk/calvin_files/parameter/src/ 
> ParameterNameValueType.o fusion_sdk/calvin_files/parsers/src/ 
> CDFFileReader.o fusion_sdk/calvin_files/parsers/src/CelFileReader.o  
> fusion_sdk/calvin_files/parsers/src/DataGroupHeaderReader.o  
> fusion_sdk/calvin_files/parsers/src/DataGroupReader.o fusion_sdk/ 
> calvin_files/parsers/src/DataSetHeaderReader.o fusion_sdk/ 
> calvin_files/parsers/src/DataSetReader.o fusion_sdk/calvin_files/ 
> parsers/src/FileHeaderReader.o fusion_sdk/calvin_files/parsers/src/ 
> FileInput.o fusion_sdk/calvin_files/parsers/src/ 
> GenericDataHeaderReader.o fusion_sdk/calvin_files/parsers/src/ 
> GenericFileReader.o fusion_sdk/calvin_files/utils/src/ 
> AffymetrixGuid.o fusion_sdk/calvin_files/utils/src/DateTime.o  
> fusion_sdk/calvin_files/utils/src/FileUtils.o fusion_sdk/calvin_files/ 
> utils/src/StringUtils.o fusion_sdk/calvin_files/utils/src/checksum.o  
> fusion_sdk/file/BPMAPFileData.o fusion_sdk/file/BPMAPFileWriter.o  
> fusion_sdk/file/CDFFileData.o fusion_sdk/file/CELFileData.o  
> fusion_sdk/file/FileIO.o fusion_sdk/file/FileWriter.o  
> R_affx_cel_parser.o R_affx_cdf_parser.o R_affx_cdf_extras.o  
> R_affx_bpmap_parser.o   -F/Library/Frameworks/R.framework/.. - 
> framework R
> ** arch - ppc
> g++-4.0 -arch ppc -isysroot /Developer/SDKs/MacOSX10.4u.sdk -no-cpp- 
> precomp -I/Library/Frameworks/R.framework/Resources/include -I/ 
> Library/Frameworks/R.framework/Resources/include/ppc -imacros  
> R_affx_constants.h -Ifusion_sdk/calvin_files/array/src -Ifusion_sdk/ 
> calvin_files/data/src -Ifusion_sdk/calvin_files/exception/src - 
> Ifusion_sdk/calvin_files/fusion/src -Ifusion_sdk/calvin_files/fusion/ 
> src/GCOSAdapter -Ifusion_sdk/calvin_files/fusion/src/CalvinAdapter - 
> Ifusion_sdk/calvin_files/parameter/src -Ifusion_sdk/calvin_files/ 
> parsers/src -Ifusion_sdk/calvin_files/portability/src -Ifusion_sdk/ 
> calvin_files/template/src -Ifusion_sdk/calvin_files/utils/src - 
> Ifusion_sdk/calvin_files/writers/src -Ifusion_sdk/file -Ifusion_sdk/ 
> portability -D_USE_MEM_MAPPING_ -I/usr/local/include    -fPIC  -g -O2  
> -Wall -O0 -c R_affx_cel_parser.cpp -o R_affx_cel_parser.o
> 
> As you see from this (and sorry for all the include statements), when  
> R starts compiling for arch = "ppc", it starts with a with in /src  
> whereas the first compilation run (or more precise the first  
> architecture) starts from /src/fusion_sdk/calvin_files/data/src. This  
> is of course because the object files in the subdirectories do not  
> get removed between the two architectures.
> 
> This has - probably - something to do with R CMD SHLIB not cleaning  
> up after itself. A method of removing $(OBJECTS) would be exactly  
> what I need, and may I suggest that this is done per default instead  
> of being set by an option. Anyone using the CRAN binary for the Mac  
> will have the same problem as I.
> 
> I am happy to see that this may be fixed for R-2.6.0. Is there  
> anything I can do in the meantime for getting my package to work  
> under R-2.5.1? (I can always move my source files from the /src/ 
> fusion_sdk subdirectories directly into /src, but I would prefer to  
> avoid that).
> 
>>> Kasper
>>>
>>> The full Makevars file:
>>>
>>> MYCXXFLAGS=-O0
>> You do realize that is highly non-portable?
> 
> Yes, but I do not know of any other way to downgrade the optimization  
> level. The SDK I am using breaks (due to memory alignment issues if a  
> specific optimization flag for GCC is include - the flag is certainly  
> implied by -O2 and as far as I recall also -O1). I guess the solution  
> is to switch to an autoconfigure script. The main reason for not  
> doing this is 1) time and actually more 2) the fact that the SDK does  
> not come with an ac script.
> 
> Thanks for the feedback, Kasper
> 
> 
>>> %.o: %.cpp
>>> 	$(CXX) $(ALL_CPPFLAGS) $(ALL_CXXFLAGS) $(MYCXXFLAGS) -c $< -o $@
>>>
>>> PKG_CPPFLAGS=\
>>> -imacros R_affx_constants.h\
>>> -Ifusion_sdk/calvin_files/array/src\
>>> -Ifusion_sdk/calvin_files/data/src\
>>> -Ifusion_sdk/calvin_files/exception/src\
>>> -Ifusion_sdk/calvin_files/fusion/src\
>>> -Ifusion_sdk/calvin_files/fusion/src/GCOSAdapter\
>>> -Ifusion_sdk/calvin_files/fusion/src/CalvinAdapter\
>>> -Ifusion_sdk/calvin_files/parameter/src\
>>> -Ifusion_sdk/calvin_files/parsers/src\
>>> -Ifusion_sdk/calvin_files/portability/src\
>>> -Ifusion_sdk/calvin_files/template/src\
>>> -Ifusion_sdk/calvin_files/utils/src\
>>> -Ifusion_sdk/calvin_files/writers/src\
>>> -Ifusion_sdk/file\
>>> -Ifusion_sdk/portability\
>>> -D_USE_MEM_MAPPING_
>>>
>>> PKG_SOURCES = \
>>> 	fusion_sdk/calvin_files/data/src/CDFData.cpp\
>>> 	fusion_sdk/calvin_files/data/src/CDFProbeGroupInformation.cpp\
>>> 	fusion_sdk/calvin_files/data/src/CDFProbeInformation.cpp\
>>> 	fusion_sdk/calvin_files/data/src/CDFProbeSetInformation.cpp\
>>> 	fusion_sdk/calvin_files/data/src/CDFQCProbeInformation.cpp\
>>> 	fusion_sdk/calvin_files/data/src/CDFQCProbeSetInformation.cpp\
>>> 	fusion_sdk/calvin_files/data/src/CELData.cpp\
>>> 	fusion_sdk/calvin_files/data/src/DataGroup.cpp\
>>> 	fusion_sdk/calvin_files/data/src/DataGroupHeader.cpp\
>>> 	fusion_sdk/calvin_files/data/src/DataSet.cpp\
>>> 	fusion_sdk/calvin_files/data/src/DataSetHeader.cpp\
>>> 	fusion_sdk/calvin_files/data/src/FileHeader.cpp\
>>> 	fusion_sdk/calvin_files/data/src/GenericData.cpp\
>>> 	fusion_sdk/calvin_files/data/src/GenericDataHeader.cpp\
>>> 	fusion_sdk/calvin_files/exception/src/ExceptionBase.cpp\
>>> 	fusion_sdk/calvin_files/fusion/src/CalvinAdapter/
>>> CalvinCELDataAdapter.cpp\
>>> 	fusion_sdk/calvin_files/fusion/src/FusionBPMAPData.cpp\
>>> 	fusion_sdk/calvin_files/fusion/src/FusionCDFData.cpp\
>>> 	fusion_sdk/calvin_files/fusion/src/FusionCDFQCProbeSetNames.cpp\
>>> 	fusion_sdk/calvin_files/fusion/src/FusionCELData.cpp\
>>> 	fusion_sdk/calvin_files/fusion/src/GCOSAdapter/ 
>>> GCOSCELDataAdapter.cpp\
>>> 	fusion_sdk/calvin_files/parameter/src/ParameterNameValueType.cpp\
>>> 	fusion_sdk/calvin_files/parsers/src/CDFFileReader.cpp\
>>> 	fusion_sdk/calvin_files/parsers/src/CelFileReader.cpp\
>>> 	fusion_sdk/calvin_files/parsers/src/DataGroupHeaderReader.cpp\
>>> 	fusion_sdk/calvin_files/parsers/src/DataGroupReader.cpp\
>>> 	fusion_sdk/calvin_files/parsers/src/DataSetHeaderReader.cpp\
>>> 	fusion_sdk/calvin_files/parsers/src/DataSetReader.cpp\
>>> 	fusion_sdk/calvin_files/parsers/src/FileHeaderReader.cpp\
>>> 	fusion_sdk/calvin_files/parsers/src/FileInput.cpp\
>>> 	fusion_sdk/calvin_files/parsers/src/GenericDataHeaderReader.cpp\
>>> 	fusion_sdk/calvin_files/parsers/src/GenericFileReader.cpp\
>>> 	fusion_sdk/calvin_files/utils/src/AffymetrixGuid.cpp\
>>> 	fusion_sdk/calvin_files/utils/src/DateTime.cpp\
>>> 	fusion_sdk/calvin_files/utils/src/FileUtils.cpp\
>>> 	fusion_sdk/calvin_files/utils/src/StringUtils.cpp\
>>> 	fusion_sdk/calvin_files/utils/src/checksum.cpp\
>>> 	fusion_sdk/file/BPMAPFileData.cpp\
>>> 	fusion_sdk/file/BPMAPFileWriter.cpp\
>>> 	fusion_sdk/file/CDFFileData.cpp\
>>> 	fusion_sdk/file/CELFileData.cpp\
>>> 	fusion_sdk/file/FileIO.cpp\
>>> 	fusion_sdk/file/FileWriter.cpp\
>>> 	R_affx_cel_parser.cpp\
>>> 	R_affx_cdf_parser.cpp\
>>> 	R_affx_cdf_extras.cpp\
>>> 	R_affx_bpmap_parser.cpp
>>>
>>> OBJS=$(PKG_SOURCES:.cpp=.o)
>>>
>>> all: $(SHLIB)
>> -- 
>> Brian D. Ripley,                  ripley at stats.ox.ac.uk
>> Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
>> University of Oxford,             Tel:  +44 1865 272861 (self)
>> 1 South Parks Road,                     +44 1865 272866 (PA)
>> Oxford OX1 3TG, UK                Fax:  +44 1865 272595
> 
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel

More information about the R-devel mailing list