[R-sig-Geo] Spatial layers for Europe at 30-m available as Cloud Optimized GeoTiffs (corrected)

Tue Mar 16 12:24:58 CET 2021

Hi Edzer thanks for spotting this.

Cloud Optimized GeoTiffs seems to be somewhat complicated when it comes 
to adding metadata etc in the tif file directly. For example, the COG 
validator reports:

# rio cogeo validate 
/data/raster/lcv/lcv_landcover.hcl_lucas.corine.rf_p_30m_0..0cm_2019_eumap_epsg3035_v0.1.tif
he following errors were found:
- The offset of the main IFD should be < 300. It is 1681443308 instead
- The offset of the IFD for overview of index 0 is 750, whereas it 
should be greater than the one of the main image, which is at byte 
1681443308
/data/raster/lcv/lcv_landcover.hcl_lucas.corine.rf_p_30m_0..0cm_2019_eumap_epsg3035_v0.1.tif 
is NOT a valid cloud optimized GeoTIFF

But it seems that the tif fails the check, but works fine on the end.

We are still only testing if this has any effect on speed of access etc. 
The worst case scenario is that users can not access the data because of 
some small glitch in the tif header.

To produce a legend for land cover maps, we recommend using:

https://gitlab.com/geoharmonizer_inea/spatial-layers/-/blob/master/map-style/lcv_landcover.hcl_lucas.corine.rf_p_30m_0..0cm_2000_eumap_epsg3035_v0.1.qml

You can of course always visually check what you get via:

https://maps.opendatascience.eu

HTH,

T. Hengl
https://opengeohub.org/about

On 3/15/21 4:40 PM, Edzer Pebesma wrote:
> Hi Tom, great work!
> 
> When reading this either with terra or stars, as in
> 
> in.tif = 
> "/vsicurl/http://s3.eu-central-1.wasabisys.com/eumap/lcv/lcv_landcover.hcl_lucas.corine.rf_p_30m_0..0cm_2019_eumap_epsg3035_v0.1.tif" 
> 
> 
> library(terra)
> # terra version 1.1.5
> plot(rast(in.tif))
> 
> #library(raster)
> #plot(raster(in.tif))
> 
> library(stars)
> # Loading required package: abind
> # Loading required package: sf
> # Linking to GEOS 3.9.0, GDAL 3.2.1, PROJ 7.2.1
> plot(read_stars(in.tif))
> 
> I see that both plots show a continuous raster, not a categorical. We 
> spent quite a bit of time recently in stars & tmap development to get 
> handling and plotting of categorical rasters right (including those 
> having a color table), but this file doesn't give proper access to the 
> categories.
> 
> The metadata tags do have them:
> 
> stars::gdal_metadata(in.tif)
> #  [1] "1=111 - Urban fabric"
> #  [2] "10=212 - Permanently irrigated arable land"
> #  [3] "11=213 - Rice fields"
> #  [4] "12=221 - Vineyards"
> #  [5] "13=222 - Fruit trees and berry plantations"
> #  [6] "14=223 - Olive groves"
> #  [7] "15=231 - Pastures"
> #  [8] "16=311 - Broad-leaved forest"
> #  [9] "17=312 - Coniferous forest"
> # [10] "18=321 - Natural grasslands"
> # [11] "19=322 - Moors and heathland"
> # [12] "2=122 - Road and rail networks and associated land"
> # [13] "20=323 - Sclerophyllous vegetation"
> # [14] "21=324 - Transitional woodland-shrub"
> # [15] "22=331 - Beaches, dunes, sands"
> # [16] "23=332 - Bare rocks"
> # [17] "24=333 - Sparsely vegetated areas"
> # [18] "25=334 - Burnt areas"
> # [19] "26=335 - Glaciers and perpetual snow"
> # [20] "27=411 - Inland wetlands"
> # [21] "28=421 - Maritime wetlands"
> # [22] "29=511 - Water courses"
> # [23] "3=123 - Port areas"
> # [24] "30=512 - Water bodies"
> # [25] "31=521 - Coastal lagoons"
> # [26] "32=522 - Estuaries"
> # [27] "33=523 - Sea and ocean"
> # [28] "4=124 - Airports"
> # [29] "5=131 - Mineral extraction sites"
> # [30] "6=132 - Dump sites"
> # [31] "7=133 - Construction sites"
> # [32] "8=141 - Green urban areas"
> # [33] "9=211 - Non-irrigated arable land"
> # [34] "AREA_OR_POINT=Area"
> 
> but GDAL doesn't give access to those in a programmatic way. I've tried 
> to add a .aux.xml file with the table, this worked locally (for both 
> stars - after using droplevels() - and terra) and might as well work 
> over the /vsicurl connection. File attached.
> 
> Many regards,
> 
> 
> On 02/03/2021 18:30, Tomislav Hengl wrote:
>>
>> We have mapped land cover classes for the 2000-2019 period for 
>> continental Europe at 30-m resolution using spatiotemporal Machine
>> Learning (we used R and python for modeling). Explore the dynamic EU 
>> landscapes on your palm using the ODS-Europe viewer: 
>> https://maps.opendatascience.eu
>>
>> To access almost 10TB of data using R you use the terra or similar
>> packages e.g.:
>>
>> R> library(terra)
>> R> in.tif = 
>> "/vsicurl/http://s3.eu-central-1.wasabisys.com/eumap/lcv/lcv_landcover.hcl_lucas.corine.rf_p_30m_0..0cm_2019_eumap_epsg3035_v0.1.tif" 
>>
>> R> tif = rast(in.tif)
>>
>>  From here you can use any native operation e.g. to crop some polygon
>> or resample / aggregate values (there is no need to download whole
>> data sets). A detailed tutorial on how to work with Cloud Optimized
>> GeoTiffs is available here: 
>> https://gitlab.com/openlandmap/global-layers/-/blob/master/tutorial/OpenLandMap_COG_tutorial.md. 
>>
>>
>> Complete list of Cloud Optimized GeoTiffs we produced so far for Europe
>> is available here: 
>> https://gitlab.com/geoharmonizer_inea/eumap/-/blob/master/gh_raster_layers.csv 
>>
>>
>> If not otherwise specified, the data available on this portal is 
>> licensed under the Open Data Commons Open Database License 
>> <https://opendatacommons.org/licenses/odbl/> (ODbL) and/or Creative
>> Commons Attribution-ShareAlike 4.0 
>> <https://creativecommons.org/licenses/by-sa/4.0/legalcode> and/or 
>> Creative Commons Attribution 4.0 
>> <https://creativecommons.org/licenses/by/4.0/legalcode> International
>> license (CC BY).
>>
>> Read more in: 
>> https://opengeohub.medium.com/europe-from-above-space-time-machine-learning-reveals-our-changing-environment-1b05cb7be520 
>>
>>
>> If you experience any technical problems or if you discover a bug,
>> please report via: 
>> https://gitlab.com/geoharmonizer_inea/spatial-layers/-/issues
>>
>> T. Hengl
>> https://opengeohub.org/about
>>
>> _______________________________________________
>> R-sig-Geo mailing list
>> R-sig-Geo using r-project.org
>> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> 
> 
> _______________________________________________
> R-sig-Geo mailing list
> R-sig-Geo using r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>