[R-pkg-devel] R CMD build hanging for some but not all packages
Martin Morgan
mtmorg@n@b|oc @end|ng |rom gm@||@com
Tue Feb 11 16:59:25 CET 2020
I don't have direct experience or responsibility for this code, but it seems like R CMD build could instead create a 'whitelist' of files to copy to the build location, and do that rather than copying everything and removing things that are to be ignored. The basis for a patch might be somewhere in the vicinity of
https://github.com/wch/r-source/blob/eb84b52444c4afff936852e11233ee84e26eaef1/src/library/tools/R/build.R#L978-L1051
I'm not sure about the reasons for the current approach, though maybe it is simpler to implement, more performant, or maybe handles edge cases (like long path names or movement across volumes?) better.
Martin Morgan
On 2/11/20, 9:31 AM, "R-package-devel on behalf of Gábor Csárdi" <r-package-devel-bounces using r-project.org on behalf of csardi.gabor using gmail.com> wrote:
It is actually rather complicated, because .Rbuildignore can have
regular expressions, that refer to files or directories, and for the
latter you need to ignore the whole directory. You can look up in the
R source code how R does it, maybe you can reuse that code.
Here is a quick and dirty function that does it, mostly based on code
from the remotes package. It is very slow for me currently, mainly
because my .git directory has thousands of files. Still faster than
copying a huge directory I guess.
Speeding it up is left as an exercise. :) I think doing a manual
filtering for /.git initially would probably make it fast enough. Oh,
it is also not super tested. :)
without_ignored <- function(path = ".",
extra = c("^\\.git$", "^\\.gitignore$")) {
paths <- dir(path, recursive = TRUE, all.files = TRUE)
ignore_file <- file.path(path, ".Rbuildignore")
ignore <- c(extra, tools:::get_exclude_patterns())
if (file.exists(ignore_file)) {
ignore <- c(ignore, readLines(ignore_file, warn = FALSE))
}
vlapply <- function(X, FUN, ..., USE.NAMES = TRUE) {
vapply(X, FUN, logical(1L), ..., USE.NAMES = USE.NAMES)
}
matches_ignores <- function(x) {
any(vlapply(ignore, grepl, x, perl = TRUE, ignore.case = TRUE))
}
directories <- function(paths) {
dirs <- unique(dirname(paths))
out <- dirs[dirs != "."]
while(length(dirs) > 0 && any(dirs != ".")) {
out <- unique(c(out, dirs[dirs != "."]))
dirs <- unique(dirname(dirs))
}
sort(out)
}
# We need to search for the paths as well as directories in the path, so
# `^foo$` matches `foo/bar`
should_ignore <- function(path) {
any(vlapply(c(path, directories(path)), matches_ignores))
}
paths[! vlapply(paths, should_ignore)]
}
Gabor
On Tue, Feb 11, 2020 at 2:23 PM Ben Bolker <bbolker using gmail.com> wrote:
>
>
> Something like file.copy( setdiff(list.files(recursive=TRUE)),
> scan(".Rbuildignore", what=character())), to = destdir) ?
>
> On 2020-02-11 8:50 a.m., Rainer Krug wrote:
> > Thinking about it - what would be the easiest way, to copy the package, excluding files in .Rbuidignore, into the temp directory?
> >
> > I would like to use directly the .Rbuildignore file, to exclude the files from copying.
> >
> > Any suggestions?
> >
> >
> >> On 11 Feb 2020, at 10:20, Rainer M Krug <Rainer using krugs.de> wrote:
> >>
> >>
> >>
> >>> On 11 Feb 2020, at 09:54, Gábor Csárdi <csardi.gabor using gmail.com> wrote:
> >>>
> >>> On Tue, Feb 11, 2020 at 8:52 AM Rainer M Krug <Rainer using krugs.de> wrote:
> >>>> On 11 Feb 2020, at 09:42, Gábor Csárdi <csardi.gabor using gmail.com> wrote:
> >>>> On Tue, Feb 11, 2020 at 8:38 AM Rainer M Krug <Rainer using krugs.de> wrote:
> >>>> On 10 Feb 2020, at 17:58, Gábor Csárdi <csardi.gabor using gmail.com> wrote:
> >>>>
> >>>> Maybe you have large, ignored files in the package directory? R first
> >>>> creates a copy of the whole directory and only applies `.Rbuildignore`
> >>>> after that.
> >>>>
> >>>> THANKS - you solved my question without me having to answer it. As I am using a makefile to build the package, I can easily rename the ignored directories before calling build.
> >>>>
> >>>> But the question is why are these ignored directories scanned before building the package? They should be ignored anyway?
> >>>>
> >>>> Rainer
> >>>>
> >>>>
> >>>> A makefile is a good idea, but I would rather move the files that are
> >>>> needed to a temporary directory, instead of temporarily renaming
> >>>> unneeded files, because if the makefile fails, then you won't restore
> >>>> them and your project will be in a broken state.
> >>>>
> >>>>
> >>>> You are right. Having two targets (prepare build, finalise build) which move the files and put them back afterwards would be the best option (and make the build call both).
> >>>>
> >>>> Is there a way to have an .onExit() type function in a makefile?
> >>>
> >>> The on-exit mechanism can fail as well. It is just better not to move
> >>> around the files of the project I think.
> >>
> >> True.
> >>
> >> Thanks,
> >>
> >> Rainer
> >>
> >>>
> >>> Gabor
> >>>
> >>> [...]
> >>
> >> --
> >> Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation Biology, UCT), Dipl. Phys. (Germany)
> >>
> >> Orcid ID: 0000-0002-7490-0066
> >>
> >> Department of Evolutionary Biology and Environmental Studies
> >> University of Zürich
> >> Office Y34-J-74
> >> Winterthurerstrasse 190
> >> 8075 Zürich
> >> Switzerland
> >>
> >> Office: +41 (0)44 635 47 64
> >> Cell: +41 (0)78 630 66 57
> >> email: Rainer.Krug using uzh.ch
> >> Rainer using krugs.de
> >> Skype: RMkrug
> >>
> >> PGP: 0x0F52F982
> >>
> >>
> >>
> >>
> >> [[alternative HTML version deleted]]
> >>
> >> ______________________________________________
> >> R-package-devel using r-project.org mailing list
> >> https://stat.ethz.ch/mailman/listinfo/r-package-devel
> >
> > --
> > Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation Biology, UCT), Dipl. Phys. (Germany)
> >
> > Orcid ID: 0000-0002-7490-0066
> >
> > Department of Evolutionary Biology and Environmental Studies
> > University of Zürich
> > Office Y34-J-74
> > Winterthurerstrasse 190
> > 8075 Zürich
> > Switzerland
> >
> > Office: +41 (0)44 635 47 64
> > Cell: +41 (0)78 630 66 57
> > email: Rainer.Krug using uzh.ch
> > Rainer using krugs.de
> > Skype: RMkrug
> >
> > PGP: 0x0F52F982
> >
> >
> >
> >
> > [[alternative HTML version deleted]]
> >
> > ______________________________________________
> > R-package-devel using r-project.org mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-package-devel
> >
>
> ______________________________________________
> R-package-devel using r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-package-devel
______________________________________________
R-package-devel using r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel
More information about the R-package-devel
mailing list