[R] RCurl: authentication when posting forms

[Valerie Obenchain]

Hi Duncan,

Yes, unfortunately my filter must have eaten your first response to my 
parseHTTPHeader question. Thanks for responding to it again here on the 
list and for the more efficient paste tip.

Below is a getURI call that returns a "not found" message. It is a 
publicly available web site and requires no authentication.

myurl <-  
"https://www.labkey.org/query/home/Study/demo/selectRows.api?schemaName=list&query.queryName=HIV%20Test%20Results"
reader <- basicTextGatherer()
header <- basicTextGatherer()
myopts <- curlOptions(writefunction=reader$update, 
headerfunction=header$update, ssl.verifyhost=FALSE, 
ssl.verifypeer=FALSE, followlocation=TRUE)
handle <- getCurlHandle()
mydata <- getURI(myurl, .opts=myopts, curl=handle)

The correct url that gives an "ok" message is
myurl <-  
"https://www.labkey.org/query/home/Study/demo/selectRows.api?schemaName=study&query.queryName=HIV%20Test%20Results"


Take care,
Valerie


Duncan Temple Lang wrote:
> Valerie Obenchain wrote:
>   
>> Duncan,
>>
>> Thank you for the examples.  I had tried all of these different options 
>> for authentication but had no luck. I was getting a "100 continue" and 
>> then a "401 unauthorized" response.  This morning the owners of the 
>> server I was trying to access discovered a bug with their api when using 
>> basic authorization. Evidently the code on their side was explicitly 
>> checking that the request method was GET, hence why all of my POST 
>> attempts were failing. They are in the process of fixing that code and 
>> my guess is that when I am able to try again the RCurl post functions 
>> will work just fine.
>>     
>
> Thanks for the update.  Good to know that there isn't a problem
> with the libcurl/RCurl code.
>
>   
>> I did have one other RCurl question I'd like to ask you about the 
>> parseHTTPHeader function.
>>
>> The parser appears to parse on spaces, so when the error message is more 
>> than one word (eg, "not found"), the message returned is "not". I have 
>> modified the parseHTTPHeader function so that it works for me. I may not 
>> have done this in the most efficient way but at least you can see what I 
>> was trying to do.
>>     
>
>
> Yes, this is a problem. Thanks for bringing it to my attention.
> I did get your mail directly to me from about 2 weeks ago 
> and I replied. Perhaps my reply got eaten by your filters.
> In it, I said that it was a bug, and that it would be
> very useful if you could send me an RCurl command
> that illustrated the error so that I could add it to the RCurl
> tests.
>  
>
> The only modification that can make your patch slightly
> "better" is that the words in the statusMessage are essentially
> all the words in the status variable, less the first two
> (the status number and the protocol name). So
>
>  header[["statusMessage"]] <- paste(els[-c(1,2)], collapse = " ")
>
> is more convenient than
>
>   hstring <- NULL
>   for(i in 3:length(els)) hstring <- paste(hstring," ",els[i],sep="")
>   hstring <- substr(hstring,2,nchar(hstring))
>   header[["statusMessage"]] <- hstring
>
>
> If you do have a URL that I can use to test the error handling code, I'd appreciate
> if you could send it to me.  
>
>
>  D.
>
>   
>> Below I have pasted my modification for the parseHTTPHeader function 
>> which I am calling parseHeader. Please let me know if you think this is 
>> a bug or if I am just using the function incorrectly.
>>
>> Thank you for the help.
>> Valerie
>>
>> #---------------------------------------------------
>> #Sample code:
>> reader <- basicTextGatherer()
>> header <- basicTextGatherer()
>> handle <- getCurlHandle()
>> myopts <- curlOptions(  netrc=1, httpheader=c(Authorization=mypwd, 
>> Accept="test/xml",
>>                                       Accept="multipart/*", 
>> 'Content-Type'="text/xml; charset=utf-8"),
>>                                       postfields=body, 
>> writefunction=reader$update, headerfunction=header$update,
>>                                       ssl.verifyhost=FALSE, 
>> ssl.verifypeer=FALSE, followlocation=TRUE)} else
>>
>> curlPerform(url=myUrl, .opts=myopts, curl=handle)
>> h <- parseHeader(header$value())
>> status <- h$status
>> message <- h$statusMessage
>> #----------------------------------------------------
>>
>> #Modified parse function:
>> parseHeader <- function (lines)
>> {
>>   if (length(lines) < 1)
>>       return(NULL)
>>   if (length(lines) == 1)
>>       lines = strsplit(lines, "\r\n")[[1]]
>>   status = lines[1]
>>   lines = lines[-c(1, length(lines))]
>>   lines = gsub("\r\n", "", lines)
>>   if (FALSE) {
>>       header = lines[-1]
>>       header <- read.dcf(textConnection(header))
>>   }
>>   else {
>>       els <- sapply(lines, function(x) strsplit(x, ":[ ]*"))
>>       header <- lapply(els, function(x) x[2])
>>       names(header) <- sapply(els, function(x) x[1])
>>   }
>>   els <- strsplit(status, " ")[[1]]
>>   header[["status"]] <- as.integer(els[2])
>> # new code below
>>   hstring <- NULL
>>   for(i in 3:length(els)) hstring <- paste(hstring," ",els[i],sep="")
>>   hstring <- substr(hstring,2,nchar(hstring))
>>   header[["statusMessage"]] <- hstring
>>   header
>> }
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> Duncan Temple Lang wrote:
>>     
>>> Hi Valerie
>>>
>>> Valerie Obenchain wrote:
>>>       
>>>> Hi,
>>>>
>>>> Has anyone successfully used RCurl for posting data to a 
>>>> password-protected site? 
>>>>         
>>> Yes. I just set up a sample form to test with and the following
>>> all work
>>>
>>>
>>> # Perl script (and HTML form for testing in the browser) taken from
>>> #   http://www.elated.com/articles/form-validation-with-perl-and-cgi/
>>>
>>>
>>> # Provide the login & password directly
>>> postForm("http://www.omegahat.org/RCurl/testPassword/form_validation.cgi", 
>>>
>>>           your_name = "Duncan",
>>>           your_age = "35-55",
>>>           your_sex = "m",
>>>           submit = "submit",
>>>           .opts = list(userpwd = "bob:welcome"))
>>>
>>> # Get the login & password in ~/.netrc
>>> postForm("http://www.omegahat.org/RCurl/testPassword/form_validation.cgi", 
>>>
>>>           your_name = "Duncan",
>>>           your_age = "35-55",
>>>           your_sex = "m",
>>>           submit = "submit",
>>>          .opts = list(netrc = TRUE))
>>>
>>> # Get the login & password from a different netrc file
>>>
>>> postForm("http://www.omegahat.org/RCurl/testPassword/form_validation.cgi", 
>>>
>>>           your_name = "Duncan",
>>>           your_age = "35-55",
>>>           your_sex = "m",
>>>           submit = "submit",
>>>           .opts = list(netrc = TRUE,
>>>                        netrc.file = 
>>> "/Users/duncan/Projects/org/omegahat/R/RCurl/inst/examples/omg.netrc"))
>>>
>>>
>>> So let me know what problems you are having and more details
>>> about the OS, version of libcurl, and a sample URL to which
>>> to post, etc.
>>>
>>>  D.
>>>
>>>       
>>>> I
>>>> have tired using option netrc=1 with both postForm and curlPerform 
>>>> (with postfields option) but can't authenticate.
>>>> I would happily provide more details if some one has had some 
>>>> experience with this.
>>>>
>>>> Thanks very much.
>>>> Valerie
>>>>
>>>> ______________________________________________
>>>> R-help at r-project.org mailing list
>>>> https://stat.ethz.ch/mailman/listinfo/r-help
>>>> PLEASE do read the posting guide 
>>>> http://www.R-project.org/posting-guide.html
>>>> and provide commented, minimal, self-contained, reproducible code.
>>>>         
>
>