[R-sig-Geo] Earthdata CMR access

[Joe Lee]

Wow, what a great start!

Unlike GES DISC OPeNDAP server,
I think the PODAAC OPeNDAP server that you’d like to access
doesn’t require Earthdata Login credentials.
That eliminates a lot of complications.

My suspicion is that your R ncdf4 package is not built with OPeNDAP support.
Can you check if your R ndf4 package work with our demo OPeNDAP server [1]?
If not, you may want to install ncdf4 with your own netCDF-4 library that enabled OPeNDAP support.

Once [1] works with your R ncdf4,
I think my posting [2] can help you to redirect URLs properly
for NASA credential-enabled OPeNDAP servers.

Good luck and thanks for sharing your code on GitHub!


[1] https://eosdap.hdfgroup.org:8080/opendap/data/hdf5/grid_1_2d.h5.dmr.html
[2] https://hdfeos.org/forums/showthread.php?t=821#4


From: Ben Tupper <btupper using bigelow.org>
Sent: Wednesday, April 13, 2022 8:59 PM
To: Joe Lee <hyoklee using hdfgroup.org>
Cc: r-sig-geo using r-project.org
Subject: Re: [R-sig-Geo] Earthdata CMR access

What I have can search for S3 resources, but I happen to focus on OPeNDAP instead.

As far as I can tell, the ncdf4 (and RNetCDF) packages don't provide utilities for establishing credentialed opendap connections.  That is handled by sort of a third party or some kind of wrapper.   Without credentials I can search, but without credentials I cannot open a connection to the opendap resources the search yields.

If I am understanding the error message shown at the bottom of the README (see below), the server is receiving the request but for lack of credentials is refusing access.  I'm stumped.

https://github.com/BigelowLab/cmr


On Wed, Apr 13, 2022 at 2:18 PM Joe Lee <hyoklee using hdfgroup.org<mailto:hyoklee using hdfgroup.org>> wrote:
>
> Yes, that's the hard part that I'm referring to.
>
> Once you pass that step, you may also want to look at another example in MATLAB [1]
> to access PODAAC data on S3 using the temporary token.
>
> I'd really appreciate your effort if you can share a working R-code project
> that can both search with CMR and then access NASA data on S3.
>
> Regards,
>
> [1] https://www.hdfeos.org/zoo/PODAAC/ATM_SURFACE_TEMP_HUM_WIND_PRES_day_mean_ECCO_V4r4_nc_c.m
>
>
> -----Original Message-----
> From: Ben Tupper <btupper using bigelow.org<mailto:btupper using bigelow.org>>
> Sent: Wednesday, April 13, 2022 1:09 PM
> To: Joe Lee <hyoklee using hdfgroup.org<mailto:hyoklee using hdfgroup.org>>
> Cc: r-sig-geo using r-project.org<mailto:r-sig-geo using r-project.org>
> Subject: Re: [R-sig-Geo] Earthdata CMR access
>
> Hi,
>
> I was planning to use the httr package or the newish httr2 package which should be able to handle the transactions.  Once a resource is tracked down via search, I plan to use the ncdf4 package to access.
>
> I can add headers to the http requests, so presumably that can include tokens.
>
> But I am finding even the simple stuff like using the API to generate a token, as shown here https://wiki.earthdata.nasa.gov/display/EL/How+to+Generate+a+User+Token
> ,  to be challenging.  Thud!
>
> Ben
>
> httr: https://CRAN.R-project.org/package=httr
> httr2: https://CRAN.R-project.org/package=httr2
> ncdf4: https://CRAN.R-project.org/package=ncdf4
>
> On Wed, Apr 13, 2022 at 1:03 PM Joe Lee <hyoklee using hdfgroup.org<mailto:hyoklee using hdfgroup.org>> wrote:
> >
> > Hi,
> >
> > I'm also interested in the solution that you're seeking since I've
> > written a Python-equivalent CMR+OPeNDAP workflow a very long time ago [1].
> >
> > It's both very tricky and difficult nowadays because NASA tries to
> > constrain AWS egress cost via Earthdata Login authentication.
> >
> > Does R provide a cURL-like library + a temporary token-based authentication?
> >
> > [1]
> > https://earthdata.nasa.gov/collaborate/open-data-services-and-software
> > /api/opendap/gesdisc-airs-opendap-cmr
> >
> >
> > -----Original Message-----
> > From: R-sig-Geo <r-sig-geo-bounces using r-project.org<mailto:r-sig-geo-bounces using r-project.org>> On Behalf Of Ben
> > Tupper
> > Sent: Wednesday, April 13, 2022 11:41 AM
> > To: r-sig-geo <r-sig-geo using r-project.org<mailto:r-sig-geo using r-project.org>>
> > Subject: [R-sig-Geo] Earthdata CMR access
> >
> > Hello,
> >
> > I am interested in accessing physical oceanography data from PO.DAAC (
> > https://earthdata.nasa.gov/eosdis/daacs/podaac) from within R using
> > opendap.  NASA is moving its data offerings to the cloud, and it is
> > adopting Common Metadata Repositories ("CMR",
> > https://cmr.earthdata.nasa.gov/search/site/docs/search/api.html) for searching.  CMR searching is in lieu of searching THREDDS catalogs.
> >
> > I'm finding the on-ramp to using CMR tricker than I had expected given the sales pitch behind CMR (or maybe the wheels have come off my car!)  I am wondering if anyone in this community has waded into using R-based CMR to access cloud-based geospatial data, and, if so, might be willing to share some insights or provide guidance?
> >
> > Regards,
> > Ben
> >
> > --
> > Ben Tupper (he/him)
> > Bigelow Laboratory for Ocean Science
> > East Boothbay, Maine
> > http://www.bigelow.org/
> > https://eco.bigelow.org
> >
> >         [[alternative HTML version deleted]]
> >
> > _______________________________________________
> > R-sig-Geo mailing list
> > R-sig-Geo using r-project.org<mailto:R-sig-Geo using r-project.org>
> > https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> >
> > _______________________________________________
> > R-sig-Geo mailing list
> > R-sig-Geo using r-project.org<mailto:R-sig-Geo using r-project.org>
> > https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>
>
>
> --
> Ben Tupper (he/him)
> Bigelow Laboratory for Ocean Science
> East Boothbay, Maine
> http://www.bigelow.org/
> https://eco.bigelow.org



--
Ben Tupper (he/him)
Bigelow Laboratory for Ocean Science
East Boothbay, Maine
http://www.bigelow.org/
https://eco.bigelow.org
--
Ben Tupper Bigelow Laboratory for Ocean Sciences https://www.bigelow.org/ Tick forecast https://eco.bigelow.org/

	[[alternative HTML version deleted]]