[R-sig-Geo] Earthdata CMR access

Ben Tupper btupper @end|ng |rom b|ge|ow@org
Thu Apr 14 15:13:37 CEST 2022 

OK - this is very helpful, Joe.  I'll report back on what I learn.

On Thu, Apr 14, 2022 at 9:05 AM Joe Lee <hyoklee using hdfgroup.org> wrote:
>
> Yes, you need to set context with session.
> I could confirm that PO.DAAC OPeNDAP server requires session.
> Thus, R ncdf4 package needs to do something extra like pydap example does.
>
> -----Original Message-----
> From: Ben Tupper <btupper using bigelow.org>
> Sent: Thursday, April 14, 2022 7:14 AM
> To: Joe Lee <hyoklee using hdfgroup.org>
> Cc: r-sig-geo using r-project.org
> Subject: Re: [R-sig-Geo] Earthdata CMR access
>
> Well, ncdf4 is a pretty good with OPeNDAP (which is one reason why we love it!)  Below is an example using your handy testing resource.
>
> Hmmm. In your example using pydap, you establish a (credentialed?) session, and then open the OPeNDAP resource within that context?  I don't think I have seen that before in R, at least not for OPeNDAP.
> I'll poke around and see if I can find something comparable.
>
>
> ### Start
> library(ncdf4)
>
> url <- "https://eosdap.hdfgroup.org:8080/opendap/data/hdf5/grid_1_2d.h5"
> x <- nc_open(url)
> library(ncdf4)
>
> url <- "https://eosdap.hdfgroup.org:8080/opendap/data/hdf5/grid_1_2d.h5"
> x <- nc_open(url)
> temp <- ncvar_get(x, varid = "temperature",
>                   start = c(2,1),
>                   count = c(2,3))
> temp
> #      [,1] [,2] [,3]
> # [1,]   10   11   12
> # [2,]   10   11   12
> ### End
>
>
> On Wed, Apr 13, 2022 at 10:25 PM Joe Lee <hyoklee using hdfgroup.org> wrote:
> >
> > Wow, what a great start!
> >
> >
> >
> > Unlike GES DISC OPeNDAP server,
> >
> > I think the PODAAC OPeNDAP server that you’d like to access
> >
> > doesn’t require Earthdata Login credentials.
> >
> > That eliminates a lot of complications.
> >
> >
> >
> > My suspicion is that your R ncdf4 package is not built with OPeNDAP support.
> >
> > Can you check if your R ndf4 package work with our demo OPeNDAP server [1]?
> >
> > If not, you may want to install ncdf4 with your own netCDF-4 library that enabled OPeNDAP support.
> >
> >
> >
> > Once [1] works with your R ncdf4,
> >
> > I think my posting [2] can help you to redirect URLs properly
> >
> > for NASA credential-enabled OPeNDAP servers.
> >
> >
> >
> > Good luck and thanks for sharing your code on GitHub!
> >
> >
> >
> >
> >
> > [1]
> > https://eosdap.hdfgroup.org:8080/opendap/data/hdf5/grid_1_2d.h5.dmr.ht
> > ml
> >
> > [2] https://hdfeos.org/forums/showthread.php?t=821#4
> >
> >
> >
> > From: Ben Tupper <btupper using bigelow.org>
> > Sent: Wednesday, April 13, 2022 8:59 PM
> > To: Joe Lee <hyoklee using hdfgroup.org>
> > Cc: r-sig-geo using r-project.org
> > Subject: Re: [R-sig-Geo] Earthdata CMR access
> >
> >
> >
> > What I have can search for S3 resources, but I happen to focus on OPeNDAP instead.
> >
> > As far as I can tell, the ncdf4 (and RNetCDF) packages don't provide utilities for establishing credentialed opendap connections.  That is handled by sort of a third party or some kind of wrapper.   Without credentials I can search, but without credentials I cannot open a connection to the opendap resources the search yields.
> >
> > If I am understanding the error message shown at the bottom of the README (see below), the server is receiving the request but for lack of credentials is refusing access.  I'm stumped.
> >
> > https://github.com/BigelowLab/cmr
> >
> >
> >
> > On Wed, Apr 13, 2022 at 2:18 PM Joe Lee <hyoklee using hdfgroup.org> wrote:
> > >
> > > Yes, that's the hard part that I'm referring to.
> > >
> > > Once you pass that step, you may also want to look at another
> > > example in MATLAB [1] to access PODAAC data on S3 using the temporary token.
> > >
> > > I'd really appreciate your effort if you can share a working R-code
> > > project that can both search with CMR and then access NASA data on S3.
> > >
> > > Regards,
> > >
> > > [1]
> > > https://www.hdfeos.org/zoo/PODAAC/ATM_SURFACE_TEMP_HUM_WIND_PRES_day
> > > _mean_ECCO_V4r4_nc_c.m
> > >
> > >
> > > -----Original Message-----
> > > From: Ben Tupper <btupper using bigelow.org>
> > > Sent: Wednesday, April 13, 2022 1:09 PM
> > > To: Joe Lee <hyoklee using hdfgroup.org>
> > > Cc: r-sig-geo using r-project.org
> > > Subject: Re: [R-sig-Geo] Earthdata CMR access
> > >
> > > Hi,
> > >
> > > I was planning to use the httr package or the newish httr2 package which should be able to handle the transactions.  Once a resource is tracked down via search, I plan to use the ncdf4 package to access.
> > >
> > > I can add headers to the http requests, so presumably that can include tokens.
> > >
> > > But I am finding even the simple stuff like using the API to
> > > generate a token, as shown here
> > > https://wiki.earthdata.nasa.gov/display/EL/How+to+Generate+a+User+To
> > > ken
> > > ,  to be challenging.  Thud!
> > >
> > > Ben
> > >
> > > httr: https://CRAN.R-project.org/package=httr
> > > httr2: https://CRAN.R-project.org/package=httr2
> > > ncdf4: https://CRAN.R-project.org/package=ncdf4
> > >
> > > On Wed, Apr 13, 2022 at 1:03 PM Joe Lee <hyoklee using hdfgroup.org> wrote:
> > > >
> > > > Hi,
> > > >
> > > > I'm also interested in the solution that you're seeking since I've
> > > > written a Python-equivalent CMR+OPeNDAP workflow a very long time ago [1].
> > > >
> > > > It's both very tricky and difficult nowadays because NASA tries to
> > > > constrain AWS egress cost via Earthdata Login authentication.
> > > >
> > > > Does R provide a cURL-like library + a temporary token-based authentication?
> > > >
> > > > [1]
> > > > https://earthdata.nasa.gov/collaborate/open-data-services-and-soft
> > > > ware /api/opendap/gesdisc-airs-opendap-cmr
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: R-sig-Geo <r-sig-geo-bounces using r-project.org> On Behalf Of Ben
> > > > Tupper
> > > > Sent: Wednesday, April 13, 2022 11:41 AM
> > > > To: r-sig-geo <r-sig-geo using r-project.org>
> > > > Subject: [R-sig-Geo] Earthdata CMR access
> > > >
> > > > Hello,
> > > >
> > > > I am interested in accessing physical oceanography data from
> > > > PO.DAAC (
> > > > https://earthdata.nasa.gov/eosdis/daacs/podaac) from within R
> > > > using opendap.  NASA is moving its data offerings to the cloud,
> > > > and it is adopting Common Metadata Repositories ("CMR",
> > > > https://cmr.earthdata.nasa.gov/search/site/docs/search/api.html) for searching.  CMR searching is in lieu of searching THREDDS catalogs.
> > > >
> > > > I'm finding the on-ramp to using CMR tricker than I had expected given the sales pitch behind CMR (or maybe the wheels have come off my car!)  I am wondering if anyone in this community has waded into using R-based CMR to access cloud-based geospatial data, and, if so, might be willing to share some insights or provide guidance?
> > > >
> > > > Regards,
> > > > Ben
> > > >
> > > > --
> > > > Ben Tupper (he/him)
> > > > Bigelow Laboratory for Ocean Science East Boothbay, Maine
> > > > http://www.bigelow.org/ https://eco.bigelow.org
> > > >
> > > >         [[alternative HTML version deleted]]
> > > >
> > > > _______________________________________________
> > > > R-sig-Geo mailing list
> > > > R-sig-Geo using r-project.org
> > > > https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> > > >
> > > > _______________________________________________
> > > > R-sig-Geo mailing list
> > > > R-sig-Geo using r-project.org
> > > > https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> > >
> > >
> > >
> > > --
> > > Ben Tupper (he/him)
> > > Bigelow Laboratory for Ocean Science East Boothbay, Maine
> > > http://www.bigelow.org/ https://eco.bigelow.org
> >
> >
> >
> > --
> > Ben Tupper (he/him)
> > Bigelow Laboratory for Ocean Science
> > East Boothbay, Maine
> > http://www.bigelow.org/
> > https://eco.bigelow.org
> >
> > --
> >
> > Ben Tupper Bigelow Laboratory for Ocean Sciences
> > https://www.bigelow.org/ Tick forecast https://eco.bigelow.org/
>
>
>
> --
> Ben Tupper (he/him)
> Bigelow Laboratory for Ocean Science
> East Boothbay, Maine
> http://www.bigelow.org/
> https://eco.bigelow.org



-- 
Ben Tupper (he/him)
Bigelow Laboratory for Ocean Science
East Boothbay, Maine
http://www.bigelow.org/
https://eco.bigelow.org

More information about the R-sig-Geo mailing list