[R] reading data from password protected url
Steven R Corsi
srcorsi at usgs.gov
Wed Jul 6 22:13:05 CEST 2011
Your method works well for my situation when I make only one call to the
database/URL with the login info. Our database is configured like the
first situation (cookies) that you described below. Now, I will need to
make multiple successive calls to get data for different sites in the
database (one call per site). It doesn't seem to work at times when I do
this. Is there something that needs to be done to re-initialize (Do I
need to log out before making the second call)?
Steven R. Corsi Phone: (608) 821-3835
Research Hydrologist email: srcorsi at usgs.gov
U.S. Geological Survey
Wisconsin Water Science Center
8505 Research Way
Middleton, WI 53562
On 6/25/2011 6:16 PM, Duncan Temple Lang wrote:
> Hi Steve
> RCurl can help you when you need to have more control over Web requests.
> The details vary from Web site to Web site and the different ways to specify
> passwords, etc.
> If the JSESSIONID and NCES_JSESSIONID are regular cookies and returned in the first
> request as cookies, then you can just have RCurl handle the cookies
> But the basics for your case are
> h = getCurlHandle( cookiefile = "")
> Then make your Web request using getURLContent(), getForm() or postForm()
> but making certain to pass the curl handle stored in h in each call, e.g.
> ans = getForm(yourURL, login = "bob", password = "jane", curl = h)
> txt = getURLContent(dataURL, curl = h)
> If JSESSIONID and NCES_JSESSIONID are not returned as cookies but HTTP header fields, then you
> need to process the header.
> Something like
> rdr = dynCurlReader(h)
> ans = getForm(yourURL, login = "bob", password = "jane", curl = h, header = rdr$update)
> Then the header from the HTTP response is available as
> and you can use parseHTTPHeader(rdr$header()) to convert it into a named vector.
> On 6/24/11 2:12 PM, Steven R Corsi wrote:
>> I am trying to retrieve data from a password protected database. I have login information and the proper url. When I
>> make a request to the url, I get back some info, but need to read the "hidden header" information that has JSESSIONID
>> and NCES_JSESSIONID. They need to be used to set cookies before sending off the actual url request that will result in
>> the data transfer. Any help would be much appreciated.
>> R-help at r-project.org mailing list
>> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
>> and provide commented, minimal, self-contained, reproducible code.
> R-help at r-project.org mailing list
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
More information about the R-help