[R-SIG-Finance] Web Scraping of SPY Stocks

[AIE ATUMA]

Dear Matt,

This worked but the Date column is presented as the last column as against the second column.

Thank You and Best Regards, 
Emeka I. Atuma
Integrity - Walk Your Talk Don't Talk Your Work






On Saturday, 26 September 2020, 19:19:28 GMT+1, Matt Cleary <mmcleary6 using gmail.com> wrote: 





Hi Emeka,

When you convert the dataset to a tibble you are losing the date values stored in the rownames. I'd recommend storing the dates externally in the function before before making the conversion then adding them back with the mutate function:

library(quantmod)
library(dplyr)
library(tibble)
library(rvest)
library(lubridate)

# Web-scrape SP500 stock list
sp_500 <- read_html("https://en.wikipedia.org/wiki/List_of_S%26P_500_companies") %>%
  html_node("table.wikitable") %>%
  html_table() %>%
  select(`Symbol`, Security, `GICS Sector`, `GICS Sub Industry`) %>%
  as_tibble()

# Format names
names(sp_500) <- sp_500 %>%
  names() %>%
  str_to_lower() %>%
  make.names()

# Show results
sp_500

get_stock_prices <- function(ticker, return_format = "tibble", ...){
  
  # Get stock prices
  stock_prices_xts <- getSymbols(Symbols = ticker, auto.assign = FALSE, ...)
  
  dates <- as.Date(rownames(as.matrix(stock_prices_xts)))
  
  # Rename
  names(stock_prices_xts) <- c("Open", "High", "Low", "Close", "Volume", "Adjusted")
  
  # Return in xts format if tibble is not specified
  if (return_format == "tibble") {
    stock_prices <- stock_prices_xts %>%
      as_tibble() %>%
      mutate(Date = dates)
  } else {
    stock_prices <- stock_prices_xts
  }
  stock_prices
}

"MA" %>%
  get_stock_prices(return_format = 'tibble') %>%
  head()

Best, 

Matt

On Sat, Sep 26, 2020 at 1:34 PM AIE ATUMA via R-SIG-Finance <r-sig-finance using r-project.org> wrote:
> Update:
> 
> The second function and the error is below:
> 
> get_stock_prices <- function(ticker), return_format = "tibble", ...) {
>   # Get stock prices
>   stock_prices_xts <- getSymbols(Symbols = ticker, auto.assign = FALSE, ...)
>   # Rename
>   names(stock_prices_xts) <- c("Open", "High", "Low", "Close", "Volume", "Adjusted")
>   # Return in xts format if tibble is not specified
>   if (return_format == "tibble") {
>     stock_prices <- stock_prices_xts %>%
>       as_tibble() %>%
>       rownames_to_column(var = "Date") %>%
>       mutate(Date = mdy(Date))
>   } else {
>     stock_prices <- stock_prices_xts
>   }
>   stock_prices
> }
> 
> "MA" %>%
>   get_stock_prices(return_format = 'tibble')%>%head()
> 
> ERROR MESSAGE:
> 
> # A tibble: 6 x 7
>   Date        Open  High   Low Close   Volume Adjusted
>   <date>     <dbl> <dbl> <dbl> <dbl>    <dbl>    <dbl>
> 1 NA          9.96  9.97  9.56  9.64 26289000     8.98
> 2 NA          9.69 10.2   9.53 10.1  27024000     9.42
> 3 NA         10.1  10.2   9.9  10.1  29632000     9.41
> 4 NA          9.90 10.2   9.9  10.1  16006000     9.41
> 5 NA         10.1  10.6  10.1  10.6  36952000     9.85
> 6 NA         10.6  10.6  10.3  10.5  35099000     9.76
> Warning message:
> All formats failed to parse. No formats found.
> 
> 
> 
> Thank You and Best Regards, 
> Emeka I. Atuma
> Integrity - Walk Your Talk Don't Talk Your Work
> 
> 
> 
> 
> 
> 
> On Saturday, 26 September 2020, 18:09:51 GMT+1, AIE ATUMA via R-SIG-Finance <r-sig-finance using r-project.org> wrote: 
> 
> 
> 
> 
> 
> Dear All,
> 
> Please I need help. I ran the below function and got the highlighted error message. How can I correct it?
> 
> library(rvest)
> library(rvest)
> library(pbapply)
> library(TTR)
> library(dygraphs)
> library(lubridate)
> library(tidyquant)
> library(timetk)
> pacman::p_load(dygraphs,DT,quantmod)
> 
> 
> # Web-scrape SP500 stock list
> sp_500 <- read_html("https://en.wikipedia.org/wiki/List_of_S%26P_500_companies") %>%
> html_node("table.wikitable") %>%
> html_table() %>%
> select(`Ticker symbol`, Security, `GICS Sector`, `GICS Sub Industry`) %>%
> as_tibble()
> # Format names
> names(sp_500) <- sp_500 %>%
> names() %>%
> str_to_lower() %>%
> make.names()
> # Show results
> sp_500
> 
> Error Message:
> 
> Error: Can't subset columns that don't exist.
> x Column `Ticker symbol` doesn't exist.
> Run `rlang::last_error()` to see where the error occurred.
> 
> 
> The second function and the error is below:
> 
> get_stock_prices <- function(ticker), return_format = "tibble", ...) {
>   # Get stock prices
>   stock_prices_xts <- getSymbols(Symbols = ticker, auto.assign = FALSE, ...)
>   # Rename
>   names(stock_prices_xts) <- c("Open", "High", "Low", "Close", "Volume", "Adjusted")
>   # Return in xts format if tibble is not specified
>   if (return_format == "tibble") {
>     stock_prices <- stock_prices_xts %>%
>       as_tibble() %>%
>       rownames_to_column(var = "Date") %>%
>       mutate(Date = mdy(Date))
>   } else {
>     stock_prices <- stock_prices_xts
>   }
>   stock_prices
> }
> 
> "MA" %>%
>   get_stock_prices(return_format = 'tibble')%>%head()
> 
> ERROR MESSAGE:
> 
> Warning message:
> All formats failed to parse. No formats found. 
> 
> 
> 
> Thank You and Best Regards, 
> Emeka I. Atuma
> Integrity - Walk Your Talk Don't Talk Your Work
> 
> 
> 
> 
> 
> 
> On Friday, 25 September 2020, 21:12:24 GMT+1, AIE ATUMA <gttga2000 using yahoo.com> wrote: 
> 
> 
> 
> 
> 
> Dear All,
> 
> Please I need help. I ran the below function and got the highlighted error message. How can I correct it?
> 
> library(rvest)
> # Web-scrape SP500 stock list
> sp_500 <- read_html("https://en.wikipedia.org/wiki/List_of_S%26P_500_companies") %>%
> html_node("table.wikitable") %>%
> html_table() %>%
> select(`Ticker symbol`, Security, `GICS Sector`, `GICS Sub Industry`) %>%
> as_tibble()
> # Format names
> names(sp_500) <- sp_500 %>%
> names() %>%
> str_to_lower() %>%
> make.names()
> # Show results
> sp_500
> 
> Error Message:
> 
> Error: Can't subset columns that don't exist.
> x Column `Ticker symbol` doesn't exist.
> Run `rlang::last_error()` to see where the error occurred.
> 
> 
> The second function and the error is below:
> 
> get_stock_prices <- function(ticker), return_format = "tibble", ...) {
>   # Get stock prices
>   stock_prices_xts <- getSymbols(Symbols = ticker, auto.assign = FALSE, ...)
>   # Rename
>   names(stock_prices_xts) <- c("Open", "High", "Low", "Close", "Volume", "Adjusted")
>   # Return in xts format if tibble is not specified
>   if (return_format == "tibble") {
>     stock_prices <- stock_prices_xts %>%
>       as_tibble() %>%
>       rownames_to_column(var = "Date") %>%
>       mutate(Date = mdy(Date))
>   } else {
>     stock_prices <- stock_prices_xts
>   }
>   stock_prices
> }
> 
> "MA" %>%
>   get_stock_prices(return_format = 'tibble')%>%head()
> 
> ERROR MESSAGE:
> 
> Warning message:
> All formats failed to parse. No formats found. 
> 
> 
> 
> Thank You and Best Regards, 
> Emeka I. Atuma
> Integrity - Walk Your Talk Don't Talk Your Work
> 
> 
> _______________________________________________
> R-SIG-Finance using r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-sig-finance
> -- Subscriber-posting only. If you want to post, subscribe first.
> -- Also note that this is not the r-help list where general R questions should go.
> 
> _______________________________________________
> R-SIG-Finance using r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-sig-finance
> -- Subscriber-posting only. If you want to post, subscribe first.
> -- Also note that this is not the r-help list where general R questions should go.
>