[Bioc-devel] error in loading pbdMPI on tokay2
Martin Morgan
martin.morgan at roswellpark.org
Thu Feb 8 22:42:56 CET 2018
On 02/08/2018 04:04 PM, Armen Abnousi wrote:
> Hi Martin,
>
> Thanks for your answer. What do you mean by "might go to the trouble of
> installing it", is there a way for us to install software on the build
> machines?
I meant that individual users might install MPI, even if it were not
installed on the build machines, so they would be able to evaluate your
code in parallel, even though the build machines evaluated it in serial.
This would be an alternative to marking your package as 'not supported
on Windows', which would mean that the user would need to go to special
measures (installing your package from source, rather than the normal
windows .zip archive).
> As for the other suggestion, I'm not sure whether using
> BiocFileCache::BiocFileCache() will be more efficient than using
> parallel apply for example returning all the results to one processor
> and then broadcasting them out to all. If every processor is to write
> its intermediate results to a file and those files are to be read by
> every other processor then there will be "n" file reads, where each file
> is possibly on a different location on the disk. Do you happen to have
> any performance evaluation results for this method. But in any case I
> did not know about BiocFileCache, thanks for sharing it.
I do not know of benchmarks. MPI would use shared memory, perhaps backed
by the file system. Serializing results to the master and back to the
worker would involve R memory management, which is not optimized for
this type of work flow. Saving to disk would provide whatever
efficiencies and guarantees offered by the OS; it seems like these would
be 'more' than R (a trivial example being serialization of very large
vectors, which will cause R to fail) and less the MPI.
Martin
>
> Thanks,
> Armen
>
> On Thu, Feb 8, 2018 at 7:58 AM, Martin Morgan
> <martin.morgan at roswellpark.org <mailto:martin.morgan at roswellpark.org>>
> wrote:
>
>
>
> On 02/07/2018 05:01 PM, Armen Abnousi wrote:
>
> Hi Valerie,
>
> Thanks for your reply. I can make it work using snow I believe.
> But the
> problem is that all functions in these packages that you have
> suggested
> (including snow; as far as I have seen) return the answer to one
> node and
> then if I want all nodes to have the same answer I will need to
> broadcast
> it to all nodes or repeat the same operation. This is very
> prohibitive,
> requiring a lot of communication. On the other hand if I could
> use MPI
> functionality, allGather would cut the operation to half (if not
> more,
> given the size of the communication that will be needed. I
> haven't been
> able to find a package with a functionality similar to allGather
> that will
> work without MPI. With larger dataset sizes, this problem
> becomes more
> threatening and that is just the opposite of what drives us to use
> parallelism. I don't know if there is any other option, maybe we
> can't do
> better, but I thought I'll let you know what my problem is and
> maybe you'll
> suggest a better way to deal with it. Thank you again.
>
>
> as a sub-optimal solution one could use the file system for data
> transfer, sending tempfile() or app-specific
> BiocFileCache::BiocFileCache() file paths between nodes; access
> could be synchronized using BiocParallel::ipc* mutex.
>
> Although the build system does not support MPI, sufficiently
> motivated users might go to the trouble of installing it; you could
> make your parallel code conditional on presence of MPI, defaulting
> to serial (maybe with an encouraging message()) otherwise. MPI
> depedencies would be moved to the Enhances: field, I think...
>
> Martin
>
>
> Thanks,
> Armen
>
> On Wed, Jan 31, 2018 at 2:37 PM, Obenchain, Valerie <
> Valerie.Obenchain at roswellpark.org
> <mailto:Valerie.Obenchain at roswellpark.org>> wrote:
>
> Hi Armen,
>
> MPI is not installed on the Windows builders so pbdMPI is
> not supported.
> Was there a specific reason you chose pbdMPI? There are
> several other
> options for parallel evaluation which are supported across
> all 3 platforms,
> e.g., BiocParallel, doParallel and batchtools.
>
> I took a brief look at what naddaR is importing from pdbMPI
> and didn't see
> anything unusual - I believe you can find all functionality
> in one of the
> alternatives. Please have a look and let us know if one of
> these will work
> for you.
>
> Thanks.
> Valerie
>
>
>
> On 01/27/2018 12:35 PM, Armen Abnousi wrote:
>
> Hi,
>
> I'm submitting a package for build, but for windows it goes
> to tokay2 and
> fails during the build with the following message:
>
> ----------------------------------------------------------------------------------
>
> * checking for file 'naddaR/DESCRIPTION' ... OK
> * preparing 'naddaR':
> * checking DESCRIPTION meta-information ... OK
> * installing the package to build vignettes
> -----------------------------------
> * installing *source* package 'naddaR' ...
> ** R
> ** inst
> ** preparing package for lazy loading
> Error : .onLoad failed in loadNamespace() for 'pbdMPI', details:
> call: inDL(x, as.logical(local), as.logical(now), ...)
> error: unable to load shared object
> 'C:/Users/pkgbuild/packagebuilder/workers/jobs/626/R-libs/pbdMPI/libs/x64/pbdMPI.dll':
> LoadLibrary failure: The specified module could not be
> found.
>
> ERROR: lazy loading failed for package 'naddaR'
> * removing
> 'C:/Users/pkgbuild/AppData/Local/Temp/Rtmp4kcgED/Rinst1ffc2775216d/naddaR'
> In R CMD INSTALL
> -----------------------------------
> ERROR: package installation failed
>
> ------------------------------------------------------------
> -----------------------
>
> How can I change this? It builds for Linux and Mac on
> Merida2 and Malbec2.
>
> Thanks,
> Armen
>
> [[alternative HTML version deleted]]
>
> _______________________________________________Bioc-devel at r-project.org
> <mailto:Bioc-devel at r-project.org> mailing
> listhttps://stat.ethz.ch/mailman/listinfo/bioc-devel
> <http://stat.ethz.ch/mailman/listinfo/bioc-devel>
>
>
>
> This email message may contain legally privileged and/or
> confidential
> information. If you are not the intended recipient(s), or
> the employee or
> agent responsible for the delivery of this message to the
> intended
> recipient(s), you are hereby notified that any disclosure,
> copying,
> distribution, or use of this email message is prohibited. If
> you have
> received this message in error, please notify the sender
> immediately by
> e-mail and delete this email message from your computer.
> Thank you.
>
>
> [[alternative HTML version deleted]]
>
> _______________________________________________
> Bioc-devel at r-project.org <mailto:Bioc-devel at r-project.org>
> mailing list
> https://stat.ethz.ch/mailman/listinfo/bioc-devel
> <https://stat.ethz.ch/mailman/listinfo/bioc-devel>
>
>
>
> This email message may contain legally privileged and/or
> confidential information. If you are not the intended recipient(s),
> or the employee or agent responsible for the delivery of this
> message to the intended recipient(s), you are hereby notified that
> any disclosure, copying, distribution, or use of this email message
> is prohibited. If you have received this message in error, please
> notify the sender immediately by e-mail and delete this email
> message from your computer. Thank you.
>
>
This email message may contain legally privileged and/or...{{dropped:2}}
More information about the Bioc-devel
mailing list