Gabriel,

Thanks for the clarification. I was avoiding depending on CodeDepends 
because I'm fairly certain that a BioC package can't depend on a package 
that isn't in either CRAN or Bioconductor. Since you point out that the 
librarySymbols code doesn't depend on any other part of the package, I 
think it would be fine to copy it into BiocParallel and use it to check 
functions for external dependencies, if that's what you're suggesting. 
Of course, we would add a comment noting that once CodeDepends makes it 
into CRAN, we should switch over to using that.

Side note 1: If we're talking about doing sanity checks on code, we 
could also check for any usage of non-local assignment ("<<-"), since we 
know that will have no effect in the subprocess, and the user might not 
expect that if they are not familiar with multi-process parallelism.

Side note 2: Your original link gave a 404 error because it had the word 
"Note" appended to it. Removing this gave a valid link: 
https://github.com/duncantl/CodeDepends/blob/forCRAN_0.3.5/R/librarySymbols.R

-Ryan


On 11/4/13, 12:13 PM, Gabriel Becker wrote:
> Ryan,
>
> I agree that in some sense it is a different problem, but my point is 
> with a different approach we can easily answer both. The code I posted 
> returns a named character vector of symbol names with package name 
> being the name.
>
> This makes it a trivial lookup to determine both a) what symbols 
> aren't available in any of the packages and b) what packages provide 
> the remaining required symbols. No extra work required.
>
> You do have to give it a list of packages to check, but it is easy to 
> write a wrapper that automatically passes it all currently attached 
> packages if desired (a combination of search() and gsub() would be a 
> quick and dirty way to do this).
>
> All that said, I'm simply trying to help. If you guys don't want to 
> use my code/approach that is your perogative as I'm not currently 
> working on BiocParallel myself.
>
> ~G
>
>
>
>
> On Mon, Nov 4, 2013 at 11:54 AM, Ryan Thompson <rct@thompsonclan.org 
> <mailto:rct@thompsonclan.org>> wrote:
>
>     The code that I wrote intentionally avoids checking for package
>     variables, since I consider that a separate problem. Package
>     variables can be provided to the child by leading the package,
>     whereas user-defined variables must be serialized in the parent
>     and sent to the child.
>
>     I think I could fairly easily adapt the same code to return a list
>     of all packages that a function depends on.
>
>     -Ryan
>
>     On Nov 4, 2013 11:35 AM, "Michael Lawrence"
>     <lawrence.michael@gene.com <mailto:lawrence.michael@gene.com>> wrote:
>
>         The dynamic nature of R limits the extent of these checks. But
>         as Ryan has
>         noted, a simple sanity check goes a long way. If what he has
>         done could be
>         extended to the rest of the search path (people always forget
>         to attach
>         packages), I think we've hit the 80% with 20%. Got a 404 on
>         that URL btw.
>
>         Michael
>
>
>         On Mon, Nov 4, 2013 at 11:05 AM, Gabriel Becker
>         <gmbecker@ucdavis.edu <mailto:gmbecker@ucdavis.edu>>wrote:
>
>         > Hey guys,
>         >
>         > Here is code that I have written which resolves library
>         names into a full
>         > list of symbols:
>         >
>         >
>         https://github.com/duncantl/CodeDepends/blob/forCRAN_0.3.5/R/librarySymbols.RNote
>         > this does not require that the packages actually be loaded
>         at the time
>         > of the check, and does not load them (or rather, it loads
>         them but does not
>         > attach them, so no searchpath muddying occurs). You do need
>         a list of
>         > packages to check though (it adds the base ones
>         automatically). It handles
>         > dependency and could be easily extended to handle suggests
>         as well I think.
>         >
>         > When CodeDepends gets pushed to cran (not my call and not
>         high on my
>         > priority list to push for currently) it will actually do
>         exactly what you
>         > want. (the forCRAN_0.3.5 branch already does and I believe it is
>         > documented, so you could use devtools to install it now).
>         >
>         > As a side note, I'm not sure that existence of a symbol is
>         sufficient (it
>         > certainly is necessary). What about situations where the
>         symbol exists but
>         > is stale compared to the value in the parent? Are we sure
>         that can never
>         > happen?
>         >
>         > ~G
>         >
>         >
>         > On Mon, Nov 4, 2013 at 7:29 AM, Michel Lang
>         <michellang@gmail.com <mailto:michellang@gmail.com>> wrote:
>         >
>         > > You might want to consider using Recall() for recursion
>         which should
>         > solve
>         > > this. Determining the required variables using heuristics
>         as codetools
>         > will
>         > > probably lead to some confusion when using functions which
>         include calls
>         > > to, e.g., with():
>         > >
>         > > f = function() {
>         > >   with(iris, Sepal.Length + Sepal.Width)
>         > > }
>         > > codetools:::findGlobals(f)
>         > >
>         > > I would suggest to write up some documentation on what the
>         function's
>         > > environment contains and how to to define variables
>         accordingly - or why
>         > it
>         > > can generally be considered a good idea to pass everything
>         essential as
>         > an
>         > > argument. Nevertheless a "bpExport" function would be a
>         good addition for
>         > > some rare corner cases in my opinion.
>         > >
>         > > Michel
>         > >
>         > >
>         > > 2013/11/3 Henrik Bengtsson <hb@biostat.ucsf.edu
>         <mailto:hb@biostat.ucsf.edu>>
>         > >
>         > > > Hi,
>         > > >
>         > > > in BiocParallel, is there a suggested (or planned) best
>         standards for
>         > > > making *locally* assigned variables (e.g. functions)
>         available to the
>         > > > applied function when it runs in a separate R process
>         (which will be
>         > > > the most common use case)?  I understand that avoid
>         local variables
>         > > > should be avoided and it's preferred to put as mush as
>         possible in
>         > > > packages, but that's not always possible or very convenient.
>         > > >
>         > > > EXAMPLE:
>         > > >
>         > > > library('BiocParallel')
>         > > > library('BatchJobs')
>         > > >
>         > > > # Here I pick a recursive functions to make the problem
>         a bit harder,
>         > > i.e.
>         > > > # the function needs to call itself ("itself" = see below)
>         > > > fib <- function(n=0) {
>         > > >   if (n < 0) stop("Invalid 'n': ", n)
>         > > >   if (n == 0 || n == 1) return(1)
>         > > >   fib(n-2) + fib(n-1)
>         > > > }
>         > > >
>         > > > # Executing in the current R session
>         > > > cluster.functions <- makeClusterFunctionsInteractive()
>         > > > bpParams <-
>         BatchJobsParam(cluster.functions=cluster.functions)
>         > > > register(bpParams)
>         > > > values <- bplapply(0:9, FUN=fib)
>         > > > ## SubmitJobs |++++++++++++++++++++++++++++++++++| 100%
>         (00:00:00)
>         > > > ## Waiting [S:0 R:0 D:10 E:0] |+++++++++++++++++++| 100%
>         (00:00:00)
>         > > >
>         > > >
>         > > > # Executing in a separate R process, where fib() is not
>         defined
>         > > > # (not specific to BiocParallel)
>         > > > cluster.functions <- makeClusterFunctionsLocal()
>         > > > bpParams <-
>         BatchJobsParam(cluster.functions=cluster.functions)
>         > > > register(bpParams)
>         > > > values <- bplapply(0:9, FUN=fib)
>         > > > ## SubmitJobs |++++++++++++++++++++++++++++++++++| 100%
>         (00:00:00)
>         > > > ## Waiting [S:0 R:0 D:10 E:0] |+++++++++++++++++++| 100%
>         (00:00:00)
>         > > > Error in LastError$store(results = results, is.error = !ok,
>         > throw.error =
>         > > > TRUE)
>         > > > :
>         > > >   Errors occurred during execution. First error message:
>         > > > Error in FUN(...): could not find function "fib"
>         > > > [...]
>         > > >
>         > > >
>         > > > # The following illustrates that the solution is not always
>         > > > straightforward.
>         > > > # (not specific to BiocParallel; must have been
>         discussed previously)
>         > > > values <- bplapply(0:9, FUN=function(n, fib) {
>         > > >   fib(n)
>         > > > }, fib=fib)
>         > > > Error in LastError$store(results = results, is.error = !ok,
>         > > > throw.error = TRUE) :
>         > > >   Errors occurred during execution. First error message:
>         > > > Error in fib(n): could not find function "fib"
>         > > > [...]
>         > > >
>         > > > # Workaround; make fib() aware of itself
>         > > > # (this is something the user need to do, and would be very
>         > > > #  hard for BiocParallel et al. to automate.  BTW,
>         should all
>         > > > #  recursive functions be implemented this way?).
>         > > > fib <- function(n=0) {
>         > > >   if (n < 0) stop("Invalid 'n': ", n)
>         > > >   if (n == 0 || n == 1) return(1)
>         > > >   fib <- sys.function() # Make function aware of itself
>         > > >   fib(n-2) + fib(n-1)
>         > > > }
>         > > > values <- bplapply(0:9, FUN=function(n, fib) {
>         > > >   fib(n)
>         > > > }, fib=fib)
>         > > >
>         > > >
>         > > > WISHLIST:
>         > > > Considering the above recursive issue solved, a slightly
>         more explicit
>         > > > and standardized solution is then:
>         > > >
>         > > > values <- bplapply(0:9, FUN=function(n, BPGLOBALS=NULL) {
>         > > >   for (name in names(BPGLOBALS)) assign(name,
>         BPGLOBALS[[name]])
>         > > >   fib(n)
>         > > > }, BPGLOBALS=list(fib=fib))
>         > > >
>         > > > Could the above be generalized into something as neat as:
>         > > >
>         > > > bpExport("fib")
>         > > > values <- bplapply(0:9, FUN=function(n) {
>         > > >   BiocParallel::bpImport("fib")
>         > > >   fib(n)
>         > > > })
>         > > >
>         > > > or ideally just (analogously to parallel::clusterExport()):
>         > > >
>         > > > bpExport("fib")
>         > > > values <- bplapply(0:9, FUN=fib)
>         > > >
>         > > > /Henrik
>         > > >
>         > > > _______________________________________________
>         > > > Bioc-devel@r-project.org
>         <mailto:Bioc-devel@r-project.org> mailing list
>         > > > https://stat.ethz.ch/mailman/listinfo/bioc-devel
>         > > >
>         > >
>         > >         [[alternative HTML version deleted]]
>         > >
>         > > _______________________________________________
>         > > Bioc-devel@r-project.org <mailto:Bioc-devel@r-project.org>
>         mailing list
>         > > https://stat.ethz.ch/mailman/listinfo/bioc-devel
>         > >
>         >
>         >
>         >
>         > --
>         > Gabriel Becker
>         > Graduate Student
>         > Statistics Department
>         > University of California, Davis
>         >
>         >         [[alternative HTML version deleted]]
>         >
>         > _______________________________________________
>         > Bioc-devel@r-project.org <mailto:Bioc-devel@r-project.org>
>         mailing list
>         > https://stat.ethz.ch/mailman/listinfo/bioc-devel
>         >
>
>                 [[alternative HTML version deleted]]
>
>         _______________________________________________
>         Bioc-devel@r-project.org <mailto:Bioc-devel@r-project.org>
>         mailing list
>         https://stat.ethz.ch/mailman/listinfo/bioc-devel
>
>
>
>
> -- 
> Gabriel Becker
> Graduate Student
> Statistics Department
> University of California, Davis


	[[alternative HTML version deleted]]

