[R-SIG-Finance] [R] RTAQ - convert function: warning causes incorrect loading of data

Sat Oct 13 17:33:03 CEST 2012

Hi Michael,

Thank you for pointing me in the right direction, I'm now using an email
client rather than Nabble.

Related to the issue I described below, it's resolved now, I have managed
to fix it myself. However, I believe this might be a bug, or at least
something that needs improving; I have described both how to reproduce this
issue and its solution in the below 4 steps:
1) library(RTAQ)
2) Create XXX_trades.csv file with the contents below using a relative
path like [somewhere]/TAQData/2010-11-01/XXX_trades.csv
SYMBOL,DATE,TIME,PRICE,SIZE,G127,CORR,COND,EX
XXX,20101101,10:30:00,11.49,500,0,0,@,B
XXX,20101101,10:30:02,11.49,322,0,0,0,B
XXX,20101101,10:30:02,11.49,178,0,0,@,B
XXX,20101101,10:30:03,11.49,500,0,0,@,B
XXX,20101101,10:30:03,11.49,187,0,0,@,D
3) 
#convert does not generate any errors/warnings, however it does not work
properly
convert(from="2010-11-01",
to="2010-11-01",datasource="[somewhere]/TAQData/",
datadestination="[somewhere]/TAQDataRData/",trades=T,quotes=F,ticker="XXX",dir=T,
extention="csv", header=T, tradecolnames=c("SYMBOL", "DATE", "TIME",
"PRICE", "SIZE", "G127", "CORR", "COND", "EX"))
#loading the RData created by convert
TAQLoad("XXX",from="2010-11-01",to="2010-11-01",datasource="[somewhere]TAQDataRData/",
trades=T,quotes=F)
#output of TAQLoad 
     SYMBOL EX  PRICE   SIZE  COND CORR G127
<NA> "XXX"  "B" "11.49" "500" "@"  "0"  "0" 
<NA> "XXX"  "B" "11.49" "322" "0"  "0"  "0" 
<NA> "XXX"  "B" "11.49" "178" "@"  "0"  "0" 
<NA> "XXX"  "B" "11.49" "500" "@"  "0"  "0" 
<NA> "XXX"  "D" "11.49" "187" "@"  "0"  "0" 
Warning message:
timezone of object (GMT) is different than current timezone (). 

Problem are the <NA>s. If one does not supply the format of date and time
to the convert function, it is assumed that the standard NYSE format is
used, and therefore RTAQ internally (convert_to_RData.r line 32) represents
this as "Y%M%D %H:%M:%S". Whilst this works fine for some things, when a
timeDate is initialised using this format (convert_to_RData.r line 102), it
does not work. timeDate expects a correct format like "%Y%m%d %H:%M:%S"
format rather than "Y%M%D %H:%M:%S".
Run the below two to confirm: 
tdobject=timeDate:::timeDate(paste(as.vector("2010-10-11"),
as.vector("10:30:30")), format="%Y%M%D
%H:%M:%S",FinCenter="GMT",zone="GMT")
#tdobject is GMT [1] [NA]
tdobject=timeDate:::timeDate(paste(as.vector("20101011"),
as.vector("10:30:30")), format="%Y%m%d
%H:%M:%S",FinCenter="GMT",zone="GMT")
#tdobject is now GMT [1] [2010-10-11 10:30:30]

Therefore, if one explicitly includes format="%Y%m%d %H:%M:%S" in the
convert function, everything works fine and the <NA> problem above is
solved; this is my solution. Can I please suggest that, once you
investigate this and provided that you confirm my understanding,
convert_to_RData.r is changed in order to use "%Y%m%d %H:%M:%S" as the
default format? 

4) My environment:
R version 2.15.1 (2012-06-22)
Platform: i686-pc-linux-gnu (32-bit)

locale:
 [1] LC_CTYPE=en_GB       LC_NUMERIC=C         LC_TIME=en_GB       
 [4] LC_COLLATE=C         LC_MONETARY=en_GB    LC_MESSAGES=en_GB   
 [7] LC_PAPER=C           LC_NAME=C            LC_ADDRESS=C        
[10] LC_TELEPHONE=C       LC_MEASUREMENT=en_GB LC_IDENTIFICATION=C 

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] RTAQ_0.2         timeDate_2160.97 xts_0.8-6        zoo_1.7-8       

loaded via a namespace (and not attached):
[1] grid_2.15.1    lattice_0.20-6

Best wishes,
Nicolae

On Fri, 12 Oct 2012 21:52:22 +0100, "R. Michael Weylandt"
<michael.weylandt at gmail.com> wrote:
> I'm forwarding this to the R-SIG-Finance list, where ou'll have a more
> specialized audience.
> 
> In the meanwhile, you may wish to look at
>
http://stackoverflow.com/questions/5963269/how-to-make-a-great-r-reproducible-example
> 
> Finally, I note you're posting from Nabble. Please do include context in
> your reply -- I don't believe Nabble does this automatically, so
> you'll need to manually include it. Most of the regular respondents on
> these lists don't use Nabble -- it is a _mailing list_ after all -- so
> we don't get the forum view you do, only emails of the individual
> posts. Combine that with the high volume of posts, and it's quite
> difficult to trace a discussion if we all don't make sure to include
> context.
> 
> Cheers,
> Michael
> 
> On Fri, Oct 12, 2012 at 7:01 PM, caprarn9 <caprarn9 at cs.man.ac.uk> wrote:
>> Hello,
>>
>> I am closely following the RTAQ documentation in order to load my
dataset
>> into R, however I get this warning when running the convert function in
>> the
>> following way:
>>
>> convert(from="2010-11-01", to="2010-11-01",datasource=datasource,
>> datadestination=datadestination,trades=T,quotes=T,ticker="BAC",dir=T,
>> extention="csv", header=T, tradecolnames=c("SYMBOL", "DATE", "TIME",
>> "PRICE", "SIZE", "G127", "CORR", "COND", "EX"),
quotecolnames=c("SYMBOL",
>> "DATE", "TIME", "BID", "OFR", "BIDSIZ", "OFRSIZ", "MODE", "EX"))
>>
>> The only warning returned is:
>> In `[<-.factor`(`*tmp*`, is.na(tdata$G127), value = c(1L, 1L, 1L,  :
>>   invalid factor level, NAs generated
>>
>> As it is a warning, the .RData files still get created and I can use
>> TAQLoad
>> to load them:
>>
>> x <-
>>
TAQLoad("BAC",from="2010-11-01",to="2010-11-01",datasource=datadestination,
>> trades=T,quotes=T)
>>
>> The PROBLEM:
>> head(x)
>>      SYMBOL EX  PRICE     SIZE    COND CORR G127
>> <NA> "BAC"  "B" "11.4900" "  500" "@"  "0"  "0"
>> ...
>>
>> This is the same for the quotes objects, but different headers
>> obviously. I
>> get a <NA> instead of the expected YYY-MM-DD HH:MM:SS format for each
>> observation.
>>
>> I've spent a fair number of hours on trying to get this right, no
>> success.
>> Can you please provide me with some guidance?
>>
>> Thank you.
>>
>> A sample from the CSV files I use:
>>
>> SYMBOL,DATE,TIME,BID,OFR,BIDSIZ,OFRSIZ,MODE,EX
>> BAC,20101101,9:30:00,11.5,11.51,5,116,12,P
>> ...
>>
>> SYMBOL,DATE,TIME,PRICE,SIZE,G127,CORR,COND,EX
>> BAC,20101101,10:30:00,11.49,500,0,0,@,B
>> ...
>>
>>
>>
>>
>> --
>> View this message in context:
>>
http://r.789695.n4.nabble.com/RTAQ-convert-function-warning-causes-incorrect-loading-of-data-tp4646025.html
>> Sent from the R help mailing list archive at Nabble.com.
>>
>> ______________________________________________
>> R-help at r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-help
>> PLEASE do read the posting guide
>> http://www.R-project.org/posting-guide.html
>> and provide commented, minimal, self-contained, reproducible code.