[R] help with web scraping
@pencer@gr@ve@ @end|ng |rom e||ect|vede|en@e@org
Fri Jul 24 15:06:44 CEST 2020
Hi Bill et al.:
That broke the dam: It gave me a character vector of length 1
consisting of 218 KB. I fed that to XML::readHTMLTable and
purrr::map_chr, both of which returned lists of 337 data.frames. The
former retained names for all the tables, absent from the latter. The
columns of the former are all character; that's not true for the latter.
Sadly, it's not quite what I want: It's one table for each
office-party combination, but it's lost the office designation. However,
I'm confident I can figure out how to hack that.
On 2020-07-23 17:46, William Michels wrote:
> Hi Spencer,
> I tried the code below on an older R-installation, and it works fine.
> Not a full solution, but it's a start:
> Loading required package: bitops
>> url <- "https://s1.sos.mo.gov/CandidatesOnWeb/DisplayCandidatesPlacement.aspx?ElectionCode=750004975"
>> M_sos <- getURL(url)
>  "\r\n<!DOCTYPE html>\r\n\r\n<html
> lang=\"en-us\">\r\n<head><title>\r\n\tSOS, Missouri - Elections:
> Offices Filed in Candidate Filing\r\n</title><meta name=\"viewport\"
> content=\"width=device-width, initial-scale=1.0\" [...remainder
> HTH, Bill.
> W. Michels, Ph.D.
> On Thu, Jul 23, 2020 at 2:55 PM Spencer Graves
> <spencer.graves using effectivedefense.org> wrote:
>> Hello, All:
>> I've failed with multiple attempts to scrape the table of
>> candidates from the website of the Missouri Secretary of State:
>> I've tried base::url, base::readLines, xml2::read_html, and
>> XML::readHTMLTable; see summary below.
>> Spencer Graves
>> sosURL <-
>> str(baseURL <- base::url(sosURL))
>> # this might give me something, but I don't know what
>> sosRead <- base::readLines(sosURL) # 404 Not Found
>> sosRb <- base::readLines(baseURL) # 404 Not Found
>> sosXml2 <- xml2::read_html(sosURL) # HTTP error 404.
>> sosXML <- XML::readHTMLTable(sosURL)
>> # List of 0; does not seem to be XML
>> R version 4.0.2 (2020-06-22)
>> Platform: x86_64-apple-darwin17.0 (64-bit)
>> Running under: macOS Catalina 10.15.5
>> Matrix products: default
>>  en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
>> attached base packages:
>>  stats graphics grDevices utils datasets
>>  methods base
>> loaded via a namespace (and not attached):
>>  compiler_4.0.2 tools_4.0.2 curl_4.3
>>  xml2_1.3.2 XML_3.99-0.3
>> R-help using r-project.org mailing list -- To UNSUBSCRIBE and more, see
>> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
>> and provide commented, minimal, self-contained, reproducible code.
More information about the R-help