[Rd] cleanup and Makevars

Kasper Daniel Hansen khansen at stat.Berkeley.EDU
Wed Jul 11 20:32:48 CEST 2007 

On Jul 11, 2007, at 9:30 AM, Prof Brian Ripley wrote:

> On Mon, 9 Jul 2007, Kasper Daniel Hansen wrote:
>
>> Hi
>>
>> This is a question prompted by the mac version of R, but as I see it,
>> it should have broader interest.
>>
>> These days the CRAN Mac binary per default compiles every package for
>> two architectures. First i386 and then ppc. In between the two
>> compilation runs, any object files in pkgname/src is removed. This
>> cleanup is necessary since otherwise Make would not recompile the
>> object files under the next architecture.
>>
>> I have a package which includes a large SDK which I have put in
>>   src/fusion_sdk
>> (and subdirectories). I take care of that by using a Makevars file
>> with essentially (the whole Makevars file is reproduced below)
>>
>> PKG_SOURCES = \
>>         fusion_sdk/calvin_files/data/src/CDFData.cpp
>>
>> OBJS=$(PKG_SOURCES:.cpp=.o)
>>
>> all: $(SHLIB)
>>
>> However, this setup does not remove the object files in the
>> fusion_sdk subdirectory. From a posting on R-sig-mac by Simon Urbanek
>> I learned that I need to clean up these directories myself.
>>
>> I have - per R extensions - attempted to do so using a pkgname/
>> cleanup script. While this works for cleaning up the sub directories
>> when I do R CMD build, it seems as if this script is not being run
>> between compilation runs (is this intentional btw.?).
>
> What do you mean by 'compilation runs'?  The 'cleanup' script will  
> be run by R CMD INSTALL --cleanup (and without --cleanup it is  
> intentionally not run).
>
>> So my question is now: how do I in a simple way clean up my
>> subdirectories? I would prefer it to be as simple as possible because
>> so far I have not really needed anything besides a Makevars file. In
>> fact the impression I have right now from looking at the SHLIB and
>> INSTALL scripts is that I might need a Makefile that removes the
>> object files _before_ starting the compilation, as I don't really see
>> any cleanup process (eg. making a target clean) - but this may very
>> well be due to my limited understanding of these scripts.
>
> I see R CMD INSTALL --clean that runs a clean up afterwards, and  
> 2.6.0 will also have R CMD INSTALL --preclean in case you forgot -- 
> clean on the last run.
>
>> Any help/hints on how to proceed? Especially if I want it to be
>> portable?
>
> What does R CMD INSTALL --clean not do that you want?
>
> If you are talking about R CMD SHLIB, that currently does not clean  
> up but this has been raised and it will most likely have a --clean  
> option in 2.6.0 that removes $(OBJECTS).

Perhaps I was not clear enough: The CRAN binary of R for Mac has two  
architectures inside $R_HOME/bin/exec, namely "ppc" and "i386". This  
means that whenever anyone installs a source package I see two  
compilation runs - the following is a shortened output:

* Installing to library '/Library/Frameworks/R.framework/Resources/ 
library'
* Installing *source* package 'affxparser' ...
** libs
** arch - i386
g++-4.0 -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk -no-cpp- 
precomp -I/Library/Frameworks/R.framework/Resources/include -I/ 
Library/Frameworks/R.framework/Resources/include/i386 -imacros  
R_affx_constants.h -Ifusion_sdk/calvin_files/array/src -Ifusion_sdk/ 
calvin_files/data/src -Ifusion_sdk/calvin_files/exception/src - 
Ifusion_sdk/calvin_files/fusion/src -Ifusion_sdk/calvin_files/fusion/ 
src/GCOSAdapter -Ifusion_sdk/calvin_files/fusion/src/CalvinAdapter - 
Ifusion_sdk/calvin_files/parameter/src -Ifusion_sdk/calvin_files/ 
parsers/src -Ifusion_sdk/calvin_files/portability/src -Ifusion_sdk/ 
calvin_files/template/src -Ifusion_sdk/calvin_files/utils/src - 
Ifusion_sdk/calvin_files/writers/src -Ifusion_sdk/file -Ifusion_sdk/ 
portability -D_USE_MEM_MAPPING_ -msse3    -fPIC  -g -O2 -Wall -O0 -c  
fusion_sdk/calvin_files/data/src/CDFData.cpp -o fusion_sdk/ 
calvin_files/data/src/CDFData.o

(Now a lot of other files gets compiled)

g++-4.0 -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk - 
dynamiclib -Wl,-macosx_version_min -Wl,10.3 -undefined dynamic_lookup  
-single_module -multiply_defined suppress -L/usr/local/lib -o  
affxparser.so fusion_sdk/calvin_files/data/src/CDFData.o fusion_sdk/ 
calvin_files/data/src/CDFProbeGroupInformation.o fusion_sdk/ 
calvin_files/data/src/CDFProbeInformation.o fusion_sdk/calvin_files/ 
data/src/CDFProbeSetInformation.o fusion_sdk/calvin_files/data/src/ 
CDFQCProbeInformation.o fusion_sdk/calvin_files/data/src/ 
CDFQCProbeSetInformation.o fusion_sdk/calvin_files/data/src/CELData.o  
fusion_sdk/calvin_files/data/src/DataGroup.o fusion_sdk/calvin_files/ 
data/src/DataGroupHeader.o fusion_sdk/calvin_files/data/src/DataSet.o  
fusion_sdk/calvin_files/data/src/DataSetHeader.o fusion_sdk/ 
calvin_files/data/src/FileHeader.o fusion_sdk/calvin_files/data/src/ 
GenericData.o fusion_sdk/calvin_files/data/src/GenericDataHeader.o  
fusion_sdk/calvin_files/exception/src/ExceptionBase.o fusion_sdk/ 
calvin_files/fusion/src/CalvinAdapter/CalvinCELDataAdapter.o  
fusion_sdk/calvin_files/fusion/src/FusionBPMAPData.o fusion_sdk/ 
calvin_files/fusion/src/FusionCDFData.o fusion_sdk/calvin_files/ 
fusion/src/FusionCDFQCProbeSetNames.o fusion_sdk/calvin_files/fusion/ 
src/FusionCELData.o fusion_sdk/calvin_files/fusion/src/GCOSAdapter/ 
GCOSCELDataAdapter.o fusion_sdk/calvin_files/parameter/src/ 
ParameterNameValueType.o fusion_sdk/calvin_files/parsers/src/ 
CDFFileReader.o fusion_sdk/calvin_files/parsers/src/CelFileReader.o  
fusion_sdk/calvin_files/parsers/src/DataGroupHeaderReader.o  
fusion_sdk/calvin_files/parsers/src/DataGroupReader.o fusion_sdk/ 
calvin_files/parsers/src/DataSetHeaderReader.o fusion_sdk/ 
calvin_files/parsers/src/DataSetReader.o fusion_sdk/calvin_files/ 
parsers/src/FileHeaderReader.o fusion_sdk/calvin_files/parsers/src/ 
FileInput.o fusion_sdk/calvin_files/parsers/src/ 
GenericDataHeaderReader.o fusion_sdk/calvin_files/parsers/src/ 
GenericFileReader.o fusion_sdk/calvin_files/utils/src/ 
AffymetrixGuid.o fusion_sdk/calvin_files/utils/src/DateTime.o  
fusion_sdk/calvin_files/utils/src/FileUtils.o fusion_sdk/calvin_files/ 
utils/src/StringUtils.o fusion_sdk/calvin_files/utils/src/checksum.o  
fusion_sdk/file/BPMAPFileData.o fusion_sdk/file/BPMAPFileWriter.o  
fusion_sdk/file/CDFFileData.o fusion_sdk/file/CELFileData.o  
fusion_sdk/file/FileIO.o fusion_sdk/file/FileWriter.o  
R_affx_cel_parser.o R_affx_cdf_parser.o R_affx_cdf_extras.o  
R_affx_bpmap_parser.o   -F/Library/Frameworks/R.framework/.. - 
framework R
** arch - ppc
g++-4.0 -arch ppc -isysroot /Developer/SDKs/MacOSX10.4u.sdk -no-cpp- 
precomp -I/Library/Frameworks/R.framework/Resources/include -I/ 
Library/Frameworks/R.framework/Resources/include/ppc -imacros  
R_affx_constants.h -Ifusion_sdk/calvin_files/array/src -Ifusion_sdk/ 
calvin_files/data/src -Ifusion_sdk/calvin_files/exception/src - 
Ifusion_sdk/calvin_files/fusion/src -Ifusion_sdk/calvin_files/fusion/ 
src/GCOSAdapter -Ifusion_sdk/calvin_files/fusion/src/CalvinAdapter - 
Ifusion_sdk/calvin_files/parameter/src -Ifusion_sdk/calvin_files/ 
parsers/src -Ifusion_sdk/calvin_files/portability/src -Ifusion_sdk/ 
calvin_files/template/src -Ifusion_sdk/calvin_files/utils/src - 
Ifusion_sdk/calvin_files/writers/src -Ifusion_sdk/file -Ifusion_sdk/ 
portability -D_USE_MEM_MAPPING_ -I/usr/local/include    -fPIC  -g -O2  
-Wall -O0 -c R_affx_cel_parser.cpp -o R_affx_cel_parser.o

As you see from this (and sorry for all the include statements), when  
R starts compiling for arch = "ppc", it starts with a with in /src  
whereas the first compilation run (or more precise the first  
architecture) starts from /src/fusion_sdk/calvin_files/data/src. This  
is of course because the object files in the subdirectories do not  
get removed between the two architectures.

This has - probably - something to do with R CMD SHLIB not cleaning  
up after itself. A method of removing $(OBJECTS) would be exactly  
what I need, and may I suggest that this is done per default instead  
of being set by an option. Anyone using the CRAN binary for the Mac  
will have the same problem as I.

I am happy to see that this may be fixed for R-2.6.0. Is there  
anything I can do in the meantime for getting my package to work  
under R-2.5.1? (I can always move my source files from the /src/ 
fusion_sdk subdirectories directly into /src, but I would prefer to  
avoid that).

>>
>> Kasper
>>
>> The full Makevars file:
>>
>> MYCXXFLAGS=-O0
>
> You do realize that is highly non-portable?

Yes, but I do not know of any other way to downgrade the optimization  
level. The SDK I am using breaks (due to memory alignment issues if a  
specific optimization flag for GCC is include - the flag is certainly  
implied by -O2 and as far as I recall also -O1). I guess the solution  
is to switch to an autoconfigure script. The main reason for not  
doing this is 1) time and actually more 2) the fact that the SDK does  
not come with an ac script.

Thanks for the feedback, Kasper


>> %.o: %.cpp
>> 	$(CXX) $(ALL_CPPFLAGS) $(ALL_CXXFLAGS) $(MYCXXFLAGS) -c $< -o $@
>>
>> PKG_CPPFLAGS=\
>> -imacros R_affx_constants.h\
>> -Ifusion_sdk/calvin_files/array/src\
>> -Ifusion_sdk/calvin_files/data/src\
>> -Ifusion_sdk/calvin_files/exception/src\
>> -Ifusion_sdk/calvin_files/fusion/src\
>> -Ifusion_sdk/calvin_files/fusion/src/GCOSAdapter\
>> -Ifusion_sdk/calvin_files/fusion/src/CalvinAdapter\
>> -Ifusion_sdk/calvin_files/parameter/src\
>> -Ifusion_sdk/calvin_files/parsers/src\
>> -Ifusion_sdk/calvin_files/portability/src\
>> -Ifusion_sdk/calvin_files/template/src\
>> -Ifusion_sdk/calvin_files/utils/src\
>> -Ifusion_sdk/calvin_files/writers/src\
>> -Ifusion_sdk/file\
>> -Ifusion_sdk/portability\
>> -D_USE_MEM_MAPPING_
>>
>> PKG_SOURCES = \
>> 	fusion_sdk/calvin_files/data/src/CDFData.cpp\
>> 	fusion_sdk/calvin_files/data/src/CDFProbeGroupInformation.cpp\
>> 	fusion_sdk/calvin_files/data/src/CDFProbeInformation.cpp\
>> 	fusion_sdk/calvin_files/data/src/CDFProbeSetInformation.cpp\
>> 	fusion_sdk/calvin_files/data/src/CDFQCProbeInformation.cpp\
>> 	fusion_sdk/calvin_files/data/src/CDFQCProbeSetInformation.cpp\
>> 	fusion_sdk/calvin_files/data/src/CELData.cpp\
>> 	fusion_sdk/calvin_files/data/src/DataGroup.cpp\
>> 	fusion_sdk/calvin_files/data/src/DataGroupHeader.cpp\
>> 	fusion_sdk/calvin_files/data/src/DataSet.cpp\
>> 	fusion_sdk/calvin_files/data/src/DataSetHeader.cpp\
>> 	fusion_sdk/calvin_files/data/src/FileHeader.cpp\
>> 	fusion_sdk/calvin_files/data/src/GenericData.cpp\
>> 	fusion_sdk/calvin_files/data/src/GenericDataHeader.cpp\
>> 	fusion_sdk/calvin_files/exception/src/ExceptionBase.cpp\
>> 	fusion_sdk/calvin_files/fusion/src/CalvinAdapter/
>> CalvinCELDataAdapter.cpp\
>> 	fusion_sdk/calvin_files/fusion/src/FusionBPMAPData.cpp\
>> 	fusion_sdk/calvin_files/fusion/src/FusionCDFData.cpp\
>> 	fusion_sdk/calvin_files/fusion/src/FusionCDFQCProbeSetNames.cpp\
>> 	fusion_sdk/calvin_files/fusion/src/FusionCELData.cpp\
>> 	fusion_sdk/calvin_files/fusion/src/GCOSAdapter/ 
>> GCOSCELDataAdapter.cpp\
>> 	fusion_sdk/calvin_files/parameter/src/ParameterNameValueType.cpp\
>> 	fusion_sdk/calvin_files/parsers/src/CDFFileReader.cpp\
>> 	fusion_sdk/calvin_files/parsers/src/CelFileReader.cpp\
>> 	fusion_sdk/calvin_files/parsers/src/DataGroupHeaderReader.cpp\
>> 	fusion_sdk/calvin_files/parsers/src/DataGroupReader.cpp\
>> 	fusion_sdk/calvin_files/parsers/src/DataSetHeaderReader.cpp\
>> 	fusion_sdk/calvin_files/parsers/src/DataSetReader.cpp\
>> 	fusion_sdk/calvin_files/parsers/src/FileHeaderReader.cpp\
>> 	fusion_sdk/calvin_files/parsers/src/FileInput.cpp\
>> 	fusion_sdk/calvin_files/parsers/src/GenericDataHeaderReader.cpp\
>> 	fusion_sdk/calvin_files/parsers/src/GenericFileReader.cpp\
>> 	fusion_sdk/calvin_files/utils/src/AffymetrixGuid.cpp\
>> 	fusion_sdk/calvin_files/utils/src/DateTime.cpp\
>> 	fusion_sdk/calvin_files/utils/src/FileUtils.cpp\
>> 	fusion_sdk/calvin_files/utils/src/StringUtils.cpp\
>> 	fusion_sdk/calvin_files/utils/src/checksum.cpp\
>> 	fusion_sdk/file/BPMAPFileData.cpp\
>> 	fusion_sdk/file/BPMAPFileWriter.cpp\
>> 	fusion_sdk/file/CDFFileData.cpp\
>> 	fusion_sdk/file/CELFileData.cpp\
>> 	fusion_sdk/file/FileIO.cpp\
>> 	fusion_sdk/file/FileWriter.cpp\
>> 	R_affx_cel_parser.cpp\
>> 	R_affx_cdf_parser.cpp\
>> 	R_affx_cdf_extras.cpp\
>> 	R_affx_bpmap_parser.cpp
>>
>> OBJS=$(PKG_SOURCES:.cpp=.o)
>>
>> all: $(SHLIB)
>
> -- 
> Brian D. Ripley,                  ripley at stats.ox.ac.uk
> Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
> University of Oxford,             Tel:  +44 1865 272861 (self)
> 1 South Parks Road,                     +44 1865 272866 (PA)
> Oxford OX1 3TG, UK                Fax:  +44 1865 272595

More information about the R-devel mailing list