[Bioc-devel] Changes to rhdf5

[Mike Smith]

That comment was more specifically aimed at some of the h5 functions e.g.
h5createDataset or h5write, that would open something internally like a
group or dataspace using H5Gopen/H5Sopen but then only run the appropriate
H5Gclose/H5Sclose at the end of the function.  If there was an error before
you got there it would leave things open as far as HDF5 was concerned, but
there was no way to access them via R.  I'm now wrapping these in on.exit()
statements so they should be sorted without user intervention.

My current outlook is that the 'high-level' h5 functions should open and
close files for the user, while with the H5 functions it's up to you to
tidy up after yourself - just like if you're using the C library.  I know
h5save doesn't obey this paradigm at the moment, but if you've got any
other examples I'm happy to take a look.

Mike

On 5 January 2018 at 18:42, Aaron Lun <alun at wehi.edu.au> wrote:

> Great work Mike; looking forward to it on BioC-devel.
>
> When are persistently open file handles generated? All the time, or on
> specific uses of the upper-case H5 functions?
>
> I ask because I recall that the HDF5 library permits multiple open file
> handles, but it throws an error if the access flags are not consistent.
> For example, when developing beachmat, I initially used H5F_ACC_TRUNC
> for opening a new file for HDF5Matrix output. However, this seemed to be
> incompatible with h5createFile(), which opened the file in H5F_ACC_RDWR.
> It took me a while to get around that.
>
> At any rate, currently beachmat works with the github version of rhdf5
> on my computer, so at least that's reassuring. And rhdf5 compiles so
> much faster now!
>
> Cheers,
>
> -A
>
> On 05/01/18 12:02, Mike Smith wrote:
> > Dear BioC Developers,
> >
> > Just a heads up to point out that I've made a fundamental change to the
> > rhdf5 package over the last week.  I've directly emailed developers of
> > packages that I think this affects, but thought I would post here too in
> > case I missed someone.
> >
> > These changes are mostly to make it link against Rhdf5lib, which in turn
> > updates the version of the HDF5 library we're using.  For the most part
> > this seems like it shouldn't disrupt things too much, but it has a really
> > dramatic effect on the H5close() function.  The previously advertised use
> > was to do some house keeping and close any HDF5 references that are left
> > open.  However if you run it with the updated version, it now shuts down
> > the HDF5 interface and you pretty much have to restart your R session for
> > any further rhdf5 commands to work.  This is obviously not ideal
> behaviour
> > if it's already incorporated into your code!
> >
> > Despite this, I'm going to leave H5close() in the package since it
> mirrors
> > part of the HDF5 API, but that may change as it feels pretty useless to a
> > package developer.  With this in mind, if you are using it I think
> there's
> > three options you can choose to update your code:
> >
> >     - There is a straight drop in replacement to close everything with
> the
> >     new h5closeAll() function.
> >     - Check if you need to close things at all.  There were a number of
> >     other functions that left open handles if they exited under error
> >     conditions that have now been updated to exit more gracefully.  If
> you were
> >     only using H5close() as a safety net it might now not be needed at
> all.
> >     - Use the appropriate close function for the HDF5 type e.g.
> H5Fclose(),
> >     H5Dclose() etc.  If you're using higher level functions like
> h5create()
> >     and h5write() this doesn't apply.
> >
> > The changes are now in the Bioc devel branch (rhdf5 version 2.23.3), and
> > should appear fairly soon.  Please let me know if I can provide any more
> > info, or things start behaving unexpectedly.  I've tried to test this
> > thoroughly, but there's always cases that I will have missed.
> >
> > Cheers,
> > Mike
> >
> >       [[alternative HTML version deleted]]
> >
> > _______________________________________________
> > Bioc-devel at r-project.org mailing list
> > https://stat.ethz.ch/mailman/listinfo/bioc-devel
> >
> _______________________________________________
> Bioc-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/bioc-devel
>

	[[alternative HTML version deleted]]