[OGRUG] RStudio, CIty of Ottawa Trees Dataset & Ottawa Tree Canopy Project.

Wed May 21 03:14:47 CEST 2014

Hi Everyone,

Since there seems to be some interest in exploring this a bit more, here's the more technical explanation of what I've done so far. I'm an R novice at best, so I would be very happy to learn of better or more efficient or more elegant ways to accomplish this. As far as a webapp, the possibility of Shiny comes to mind, but I have no idea whether that's something you could bundle up and embed on a webpage and/or whether your Shiny server and R implementation necessarily need to be on the same machine. 

The dataset is here: http://data.ottawa.ca/dataset/tree-inventory-street-trees

The KML file is basically just a flavour of XML. I spent probably too much time messing around with the R XML stuff before finding the sp and rgdal packages (both at CRAN). The readOGR function from rgdal gave me a SpatialPointsDataframe object, that took just over half the size of the KML file in RAM. This gets you the coordinates in an array, and metadata still stored in blobs of XML. At first I just used perl regular expressions to go fishing for the fields I wanted but later I used readHTMLTable and laply to grab everything. 

Then to get it all into a pretty plot, I used ggmap to put a map down and ggplot to put the trees on top. At the end you get something like the linked PDF from the R script I included below:

https://dl.dropboxusercontent.com/u/4647535/GlebeTrees.pdf

Cheers,

Dan Buijs

On 2014-05-20, at 8:55 PM, Joseph Potvin wrote:

> Thanks Tyler. Sounds like a great agenda item for the OGRUG meeting
> that David Bissessar is talking about organizing.
> 
> Joseph Potvin
> 
> On Tue, May 20, 2014 at 8:03 PM, Tyler Smith <tyler at plantarum.ca> wrote:
>> Hi Joseph,
>> 
>> I am a botanist at Agriculture Canada, and also a reasonably experienced R
>> user. I am very interested in this project, and would be happy to contribute
>> a either a plant biologist, an R coder or both.
>> 
>> At present I am in the field, and won't be back to Ottawa until the
>> beginning of June.
>> 
>> Best,
>> 
>> Tyler
>> 
>> On 17/05/14 06:39 PM, Joseph Potvin wrote:
>>> 
>>> Anyone in the OGRUG community have a bit of time & expertise to share
>>> towards advancing data-driven analysis of Ottawa's trees using R?
>>> Some current discussion is forwarded below.
>>> 
>>> Note to Daniel and Erwin: OGRUG is the Ottawa-Gatineau R Users Group.
>>> 
>>> Joseph Potvin
>>> 
>>> ---------- Forwarded message ----------
>>> From: Daniel Buijs <dbuijs at gmail.com>
>>> Date: Sat, May 17, 2014 at 5:48 PM
>>> Subject: Re: CIty of Ottawa Trees dataset
>>> To: Erwin Dreessen <erwin_dreessen at ncf.ca>
>>> Cc: "J. P. Unger" <ungerjp at yahoo.com>, Amy Kempster
>>> <Amykempster at bell.net>, "Nicole Desroches (home)"
>>> <nicolepdesroches at gmail.com>, Sol Shuster <sshuster at magma.ca>,
>>> elina.here at hotmail.com, Jason Kania <jason.kania at ymail.com>, Joseph
>>> Potvin <jpotvin at opman.ca>
>>> 
>>> 
>>> Hi Erwin,
>>> 
>>> It would take a bit of work to turn this into an app, and I'm afraid
>>> that's a skill set that I don't have at the moment ;) I would be very
>>> happy to chat with someone technically inclined who would be
>>> interested in turning it into an app, and I could perhaps suggest some
>>> more or less straightforward ways to do that.
>>> 
>>> The file that the city published is in a format that can be opened by
>>> Google Earth, but it was just so massive that I stopped waiting for it
>>> after an hour or so on my little laptop, so I figured out how to slice
>>> it up into smaller pieces with the free software package RStudio
>>> (www.rstudio.org). On the most basic level, I can take the 200 MB kml
>>> file that's on the City's website and turn it into a simple
>>> spreadsheet table that should be fairly straightforward to use in
>>> things like Google Maps. The bells and whistles, in terms of
>>> colour-coding by species, making the dot size represent tree diameter,
>>> and overlaying Xs on the ones that are being treated for Emerald Ash
>>> borer are things I figured out how to do in R for static PDF
>>> documents. I'm sure you could do things like this in a web app, I just
>>> don't know how.
>>> 
>>> As far as the data itself, my understanding of what the City published
>>> is that it's an extract of their internal database that the forestry
>>> department uses anyway, so I don't think they have any dedicated
>>> resources to publishing it or have committed to any particular update
>>> frequency. The date on the dataset is April 2013. I would certainly
>>> like to see them update it on some sort of predictable schedule, but
>>> my experience with open data at the moment is that you take what you
>>> can get and don't ask too many questions. :)
>>> 
>>> The City indicates that the dataset includes only city-owned street
>>> trees, so no private trees, and none of the city trees in parks or
>>> other greenspace. The data they have included is the GPS co-ordinates
>>> (which seem to line up with Google maps without any further
>>> adjustment, at least in my neighbourhood), the species of tree,
>>> diameter at breast height (no units given, but I'm guessing in cm),
>>> Ward, Street Number and Name, location of the tree in relation to the
>>> property, location of street trees that aren't associated with a civic
>>> address (i.e. medians, etc.), whether the tree is being treated for
>>> dutch elm disease or emerald ash borer, and the tree tag number for
>>> trees that are being treated.
>>> 
>>> No, I'm not working with Kevin. I've only just seen his website, with
>>> your note below. I'm certainly happy to share what I've learned with
>>> anyone who's interested. I'm afraid I have no particular intent or
>>> agenda with this, I really just wanted to find out what kind of tree
>>> was kitty-corner to my house and whether the City was going to treat
>>> it for Emerald Ash Borer. In retrospect, there were perhaps less
>>> onerous ways to do this ;).
>>> 
>>> Cheers,
>>> 
>>> Dan
>>> 
>>> 
>>> On 2014-05-17, at 3:15 PM, Erwin Dreessen wrote:
>>> 
>>> Hello Daniel Buijs,
>>> 
>>> Thanks for this.  I knew the City was brewing up various data about
>>> the city-owned trees.  You're way ahead of me on the technology, but
>>> the example you gave, adapting the City's raw data, looks terrific.
>>> If this app could be made available via our web site, that would be
>>> excellent.  Jason Kania is your contact if this requires anything
>>> special.
>>> 
>>> Do you believe the City is committed to keeping the data current?  Any
>>> idea what percent of its resources is going to this?
>>> 
>>> The Tree Canopy project, as I understand it, makes no distinction
>>> between public and private trees, nor does it distinguish species.  It
>>> is only concerned with tree canopy cover.
>>> 
>>> Are you working with Kevin O'Donnell?  (http://ottwatch.ca/)  It would
>>> be good to find your app there.
>>> 
>>> Regards,
>>> 
>>> Erwin
>>> cc: Board, Joseph
>>> 
>>> From: J. P. Unger [mailto:ungerjp at yahoo.com]
>>> Sent: Friday, May 16, 2014 10:59 PM
>>> To: Erwin Dreessen; Amy Kempster; Nicole Desroches (home); Sol
>>> Shuster; elina.here at hotmail.com; 'Jason Kania'
>>> Subject: Fw: CIty of Ottawa Trees dataset
>>> 
>>> Hi!
>>> This came in...
>>> I sent a quick reply with thanks, saying I'd forward it to the chair &
>>> directors (cc'ing Joseph Potvin).
>>> JP
>>> 
>>> ----- Forwarded Message -----
>>> From: Daniel Buijs <dbuijs at gmail.com>
>>> To: contact at greenspace-alliance.ca
>>> Cc: jpotvin at opman.ca
>>> Sent: Friday, May 16, 2014 9:43:15 PM
>>> Subject: CIty of Ottawa Trees dataset
>>> 
>>> 
>>> Hi,
>>> 
>>> Joseph Potvin forwarded a note about your upcoming meeting where you
>>> will discuss the Ottawa Tree Canopy project. I thought you might find
>>> it helpful to know (if you didn't already) that the City of Ottawa has
>>> published a dataset of City-Owned street trees, available here:
>>> 
>>> data.ottawa.ca
>>> 
>>> It's available as a kml file. I found the kml file a bit too unwieldy,
>>> so I wrote an R script that can extract the data into a more useful
>>> format and intelligible plot. The plot below shows the trees of the
>>> Glebe, colour coded by species, with dot sizes representing diameter.
>>> Trees marked with an X have been treated for Emerald Ash Borer.
>>> 
>>> If this is useful and/or of interest to you or your group, I can share
>>> the scripts I used to extract the data and to produce the plot below.
>>> 
>>> Daniel Buijs
>>> 
>>> 
>>> 
>>> ---------- Forwarded message ----------
>>> From: Erwin Dreessen <erwin_dreessen at ncf.ca>
>>> Date: Thu, May 15, 2014 at 5:50 PM
>>> Subject: Greenspace Alliance AGM - Thursday, May 29, 7 p.m.
>>> To: green-news at greenspace-alliance.ca,
>>> 
>>> 
>>> Hello Members and Friends,
>>> 
>>> You are heartily invited to the Greenspace Alliance's Annual General
>>> Meeting on Thursday, May 29, 2014, starting at 7 p.m. at the
>>> Hintonburg Community Centre, 1064 Wellington Street West (Laurel
>>> Room).
>>> 
>>> Our feature presentation will be on Phase 1 of the Ottawa Tree Canopy
>>> project.  Unfortunately, Regie Alam had to cancel but the project's
>>> co-leader, Eric Jones, will present in his stead, assisted by Iola
>>> Price.  This is a citizen science project that is estimating the tree
>>> canopy within the urban boundary on a ward basis, using open data and
>>> open source tools.
>>> 
>>> The agenda is below.  Come early (6:45 p.m.) to renew your membership or
>>> apply!
>>> 
>>> A brief General Meeting will follow the AGM.
>>> 
>>> Hope to see many of you there.
>>> 
>>> Erwin
>>> 
>>> _______________________________________________
>>> R-UG-Ottawa mailing list
>>> R-UG-Ottawa at r-project.org
>>> https://stat.ethz.ch/mailman/listinfo/r-ug-ottawa
>>> 
>> 
> 
> 
> 
> -- 
> Joseph Potvin
> Operations Manager | Gestionnaire des opérations
> The Opman Company | La compagnie Opman
> jpotvin at opman.ca
> Mobile: 819-593-5983

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://stat.ethz.ch/pipermail/r-ug-ottawa/attachments/20140520/0c47cf4b/attachment-0002.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OttawaTrees.R
Type: application/octet-stream
Size: 1724 bytes
Desc: not available
URL: <https://stat.ethz.ch/pipermail/r-ug-ottawa/attachments/20140520/0c47cf4b/attachment-0001.obj>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://stat.ethz.ch/pipermail/r-ug-ottawa/attachments/20140520/0c47cf4b/attachment-0003.html>