[R] Fetching data

[Barry Rowlingson]

On Mon, Nov 15, 2010 at 3:46 PM, Feng Mai <maifeng at gmail.com> wrote:
>
> IMO it is not possible. The code behind aspx page queries data from a
> database server and display it on the webpage.

 That doesn't make it possible. Your web browser is sending a request
to the web server, and whatever happens behind the scenes at that end,
what comes back is a table in a web page which can be scraped.

 The tricky thing is constructing exactly the right request to get the
data you want.

 It can sometimes be as simple as constructing a url, something like
'http://example.com/price/gold/2010/11/12", or it could need
parameters: "http://example.com/price/commodity=gold&year=2010&month=11".

 Or it can be like this - some active server pages that pass a massive
VIEWSTATE object back and forth on each request. It's not impossible
to script it, but you just have a lot more mess to deal with. I have
done this kind of thing for some stupid web sites in the past. These
days people enjoy building sensible APIs to their web data.

 Oh, and it might be violating the site's terms and conditions of course.

Barry