[R] Opening or activating a URL to access data, alternative to browseURL
utz.ryan at gmail.com
Tue Oct 11 13:59:50 CEST 2016
Thanks for writing. I think some of the things Bob mentioned might work,
but I'm still not quite getting there. Below is the example I'm working
# This opens the URL and creates a link to machine-readable data on the
page, which I can then download by simply doing this:
#This is what I need to read in terms of data, but this URL only exists if
the URL ran above is activated first
So, for example, try running line #2 without the first line- it won't work.
Next run #1 then #2- works fine.
See what I mean?
On Thu, Sep 29, 2016 at 5:09 PM, Bob Rudis <bob at rud.is> wrote:
> The rvest/httr/curl trio can do the cookie management pretty well. Make
> the initial connection via rvest::html_session() and then hopefully be able
> to use other rvest function calls, but curl and httr calls will use the
> cached in-memory handle info seamlessly. You'd need to store and retrieve
> cookies if you need them preserved between R sessions.
> Failing the above and assuming this would not need to be lightning fast,
> use the phantomjs or firefox web driver (either with RSelenium or some new
> stuff rOpenSci is cooking up) which will then do what browsers do best and
> maintain all this state for you. You can still slurp the page contents up
> with xml2::read_html() and use the super handy processing idioms in the
> scraping tidyverse (it needs it's own name).
> A concrete example (assuming the URLs aren't sensitive) would enable me or
> someone else to mock up something for you.
> On Thu, Sep 29, 2016 at 4:59 PM, Duncan Murdoch <murdoch.duncan at gmail.com>
>> On 29/09/2016 3:29 PM, Ryan Utz wrote:
>>> Hi all,
>>> I've got a situation that involves activating a URL so that a link to
>>> data becomes available for download. I can easily use 'browseURL' to do
>>> but I'm hoping to make this batch-process-able, and I would prefer to not
>>> have 100s of browser windows open when I go to download multiple data
>>> Here's the example:
>>> # This opens the URL and creates a link to machine-readable data on the
>>> page, which I can then download by simply doing this:
>>> However, I can only get the second line above to work if the thing in
>>> #1 has been opened in a browser already. Is there any way to allow me to
>>> either 1) close the browser after it's been opened or 2) execute the line
>>> #2 above without having to open a browser? We have hundreds of species
>>> you can see after the '&kind=' bit of the URL, so I'm trying to keep the
>>> browsing situation sane.
>> You'll need to figure out what happens when you open the first page. Does
>> it set a cookie? Does it record your IP address? Does it just build the
>> file but record nothing about you?
>> If it's one of the simpler versions, you can just read the first page,
>> wait a bit, then read the second one.
>> If you need to manage cookies, you'll need something more complicated. I
>> don't know the easiest way to do that.
>> Duncan Murdoch
>> R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
>> PLEASE do read the posting guide http://www.R-project.org/posti
>> and provide commented, minimal, self-contained, reproducible code.
Ryan Utz, Ph.D.
Assistant professor of water resources
Home/Cell: (724) 272-7769
[[alternative HTML version deleted]]
More information about the R-help