[R] XML and RCurl: problem with encoding (htmlTreeParse)

[Duncan Temple Lang]

Hi Ryusuke

 I would use the encoding parameter of htmlParse() and 
 download and parse the content in one operation:

     htmlParse("http://home.sina.com", encoding = "UTF-8")

 If you want to use getURL() in RCurl, use the .encoding parameter

  You didn't tell us the output of Sys.getlocale()
  or how your terminal/console is configured, so the above
  may vary under your configuration, but works on various
  machines for me with different settings.

    D.


Ryusuke Kenji wrote:
> 
> Hi All,
> 
> First method:-
> >library(XML)
> 
> >theurl <- "http://home.sina.com"
> >download.file(theurl, "tmp.html")
> 
> >txt <- readLines("tmp.html")
> 
> >txt <- htmlTreeParse(txt, error=function(...){}, useInternalNodes = 
> TRUE)
> 
> >g <- xpathSApply(txt, "//p", function(x) xmlValue(x))
> 
> >head(grep(" ", g, value=T))
> 
> 
> [1] "?????? | ?????? | ENGLISH"                               "??????????????? ???????????????"                        
> [3] "??????? ?????????? ??????????????????(???)"              "?????????????????????????????? ????????????????????????"
> [5] " ???????????????????????????????????????"                "? ??????????! ????? ??????! ????????????????????????!"  
> 
> 
> 
> SecondMethod:-
> >library(RCurl)
> 
> >theurl <- getURL("http://home.sina.com",encoding='GB2312')
> 
> >Encoding(theurl)
> 
> [1]"unknown"
> 
> >txt <- readLines(con=textConnection(theurl),encoding='GB2312')
> >txt[5:10] #show the lines which occurred encoding problem.
> [1] "<meta http-equiv=\"Content-Type\" content=\"text/html; charset=utf-8\" />"
> [2] "<title>SINA.com US ????????? -??????</title>"
> [3] "<meta name=\"Keywords\" content=\"????????????, ???????????????, ???????????????, ??????????????????,????????????, SINA, US, News, Chinese, Asia\" />"
> [4] "<meta name=\"Description\" content=\"???????????????????????????????????????, ???????????????24????????????????????????????????, ????????????????????????, ????????????, ??????????????????, ????????????????????????, ?????????BBS, ???????????????????????????????????.\" />"
> [5]""                                                                                                                                                                                                                                                                                
> [6] "<link rel=\"stylesheet\" type=\"text/css\" href=\"http://ui.sina.com/assets/css/style_home.css\" />"
> 
> i am trying to read data from a Chinese language website, but the Chinese characters always unreadable, may I know if any good idea to cope such encoding problem in RCurl and XML?
> 
> 
> Regards,
> Ryusuke
>  		 	   		  
> _________________________________________________________________
> 
> 
> 	[[alternative HTML version deleted]]
> 

> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.


-- 
"There are men who can think no deeper than a fact" - Voltaire


Duncan Temple Lang                duncan at wald.ucdavis.edu
Department of Statistics          work:  (530) 752-4782
4210 Mathematical Sciences Bldg.  fax:   (530) 752-7099
One Shields Ave.
University of California at Davis
Davis, CA 95616, USA



-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: <https://stat.ethz.ch/pipermail/r-help/attachments/20100703/3120b3e5/attachment.bin>