[R-pkg-devel] robust download function in R (similar to wget)?

[Bob Rudis]

libcurl (which the RCurl & curl packages are built on) do not inherently
have retry or resume partial capabilities, but those could be packaged up
into a "robustdownloader" package. There's no guarantee of wget or curl
binaries being on a system (especially Windows, even with an Rtools
installation, rly great platform there). Windows does have a PowerShell
`wget` alias for `Invoke-WebRequest` but it's woefully less powerful than
wget-proper and doesn't support the features you need and it relies on
PowerShell).

I'd suggest the creation of a separate package for robust downloading and
then relying on that or keep the solution you've got by PR #56.

On Thu, Aug 25, 2016 at 9:56 AM, Dirk Eddelbuettel <edd at debian.org> wrote:

>
> On 25 August 2016 at 08:24, Joshua Ulrich wrote:
> | Try downloader: https://cran.r-project.org/package=downloader
>
> Also, if you really want wget / know how to parameterise wget you can
> ... just depend on _wget itself_ which you can
>
> a) test for via Sys.which("wget")
>
> b) select as an option for download.file() and hence all file access.
>
> Dirk
>
> | On Thu, Aug 25, 2016 at 7:47 AM, Lucas Ferreira Mation
> | <lucasmation at gmail.com> wrote:
> | > I am creating this package <https://github.com/
> lucasmation/microdadosBrasil>
> | > that has to download data from some sources that are a bit
> | > unstable/unreliable. The problem is described in this issue
> | > <https://github.com/lucasmation/microdadosBrasil/issues/56>.
> | >
> | > Is there a more "robust" download fucntion in R?
> | > Something similar to wget, that would:
> | >
> | > - where there are errors, repeat the download a few times before
> giving up,
> | > preferably, using partial data on a file being downloaded from previous
> | > attempts
> | > - check the integrity of the downloaded file
> | >
> | > As this has to be embedded in an R package, the solution needs to be OS
> | > independent and require no installation of external software by the
> user.
> | >
> | > regards
> | > Lucas
> | >
> | >         [[alternative HTML version deleted]]
> | >
> | > ______________________________________________
> | > R-package-devel at r-project.org mailing list
> | > https://stat.ethz.ch/mailman/listinfo/r-package-devel
> |
> |
> |
> | --
> | Joshua Ulrich  |  about.me/joshuaulrich
> | FOSS Trading  |  www.fosstrading.com
> | R/Finance 2016 | www.rinfinance.com
> |
> | ______________________________________________
> | R-package-devel at r-project.org mailing list
> | https://stat.ethz.ch/mailman/listinfo/r-package-devel
>
> --
> http://dirk.eddelbuettel.com | @eddelbuettel | edd at debian.org
>
> ______________________________________________
> R-package-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-package-devel
>

	[[alternative HTML version deleted]]