[Bioc-devel] Error in HDF5 - Package submission - Not detected locally

[Pages, Herve]

Hi Tiago,

On 9/16/19 06:35, Tiago Lubiana Alves wrote:
> Hello Mike,
> 
> Thank you for the detailed explanation.
> 
> You are right, for the vignette, I can download it from the ExperimentHub
> subset the pbmc3k dataset in the first few lines. The main point of having
> a new dataset was to use it in the examples of functions. The full dataset
> is too big and the examples would take too long otherwise.
> 
> Still, if someone knows how to "transform from an HDF5 back
> SingleCellExperiment to using the data in memory", it would be of great
> help.

Just to clarify, it's not that you have to choose between using the full 
on-disk dataset or using a small in-memory subset in your vignette. A 
3rd option is to start your vignette with the full on-disk dataset and 
to immediately subset it. Like what Pete is doing in his 3-line snippet 
but without the last line (replacement of on-disk assay with in-memory 
assay).

H.


> Best,
> Tiago
> 
> 
> Tiago Lubiana
> Mestrando em Bioinformática, FCF/USP
> *Computational Systems Biology Laboratory (CSBL)*
> Telefone (laboratório): +55 (11) 2648-0240
> Telefone (pessoal): +55 (11) 954258000
> 
> 
> On Mon, Sep 16, 2019 at 5:39 AM Mike Smith <grimbough using gmail.com> wrote:
> 
>> Hi Tiago,
>>
>> I suspect what has happened here is that when create the mini_pbmc3k
>> object, you're doing this by subsetting the PBMC, 3k scRNA-seq data from
>> ExperimenHub. The assay data for that are are actually stored in an HDF5
>> file which will be downloaded and stored in your ExperimentHub cache on
>> your local machine. When you take the subset for your new object it still
>> points to the same HDF5 file, and when you save mini_pbmc3k all it actually
>> saves is the location of the HDF5 rather than putting the data in the
>> object. This works locally, but naturally the BioC build server doesn't
>> have the original HDF5 in exactly the same location, and so it fails.
>>
>> Is there a reason you don't want to use the whole dataset and make the
>> first section of the vignette demonstrate how to download it via
>> ExperimentHub? Alternatively there is a way to transform from an HDF5 back
>> SingleCellExperiment to using the data in memory, but I can't remember it
>> right now - hopefully someone else will be along shortly.
>>
>> This also seems like the perfect opportunity to point out that our next
>> BioC Developers' Forum (
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__stat.ethz.ch_pipermail_bioc-2Ddevel_2019-2DSeptember_015499.html&d=DwIFaQ&c=eRAMFD45gAfqt84VtBcfhQ&r=BK7q3XeAvimeWdGbWY_wJYbW0WYiZvSXAJJKaaPhzWA&m=s7A_Q0fNE8fpu3Ob4M_wxwPfLdlY2QkGSbNwInWB2mo&s=TDo6rxwdtAE-kwlyPoYbbHn9JI69Wdw3scAgxje60ck&e= ) will
>> have a discussion on object serilaisation which should cover exactly this
>> type of issue.
>>
>> Best,
>> Mike
>>
>>
>>
>> On Sat, 14 Sep 2019 at 23:26, Tiago Lubiana Alves <
>> tiago.lubiana.alves using usp.br> wrote:
>>
>>> Hello,
>>>
>>> I am having a problem with a package submission build.
>>>
>>> This is the package issue:
>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_Bioconductor_Contributions_issues_1241&d=DwIFaQ&c=eRAMFD45gAfqt84VtBcfhQ&r=BK7q3XeAvimeWdGbWY_wJYbW0WYiZvSXAJJKaaPhzWA&m=s7A_Q0fNE8fpu3Ob4M_wxwPfLdlY2QkGSbNwInWB2mo&s=HRcbdQHP2-fSp-0NnFZw8H0NrGgT8IerVORN96IaUrs&e=
>>> And this is the ERROR:
>>>
>>> HDF5-DIAG: Error detected in HDF5 (1.10.5) thread 0:
>>>    #000: C:/hdf5_build/CMake-hdf5-1.10.5/hdf5-1.10.5/src/H5F.c line 509
>>> in H5Fopen(): unable to open file
>>>      major: File accessibilty
>>>      minor: Unable to open file
>>>    #001: C:/hdf5_build/CMake-hdf5-1.10.5/hdf5-1.10.5/src/H5Fint.c line
>>> 1498 in H5F_open(): unable to open file: time = Fri Sep 13 15:05:35
>>> 2019
>>> , name = '/home/lubianat/.cache/ExperimentHub/5486ffbe0e3_1605',
>>> tent_flags = 0
>>> (...)
>>>
>>> Quitting from lines 46-51 (fcoex.Rmd)
>>> Error: processing vignette 'fcoex.Rmd' failed with diagnostics:
>>> failed to open file '/home/lubianat/.cache/ExperimentHub/5486ffbe0e3_1605'
>>> --- failed re-building 'fcoex.Rmd'
>>>
>>>
>>> It looks like something is pointing in the wrong direction, but I have not
>>> been able to figure out exactly what. I've tried re-saving the data file
>>> ("mini_pbmc3k"), which is loaded in the vignette chunk of the error, but
>>> that did not help.
>>>
>>> Does anyone have a suggestion of what might be happening?
>>>
>>> Thank you very much for your time,
>>> Best,
>>> Tiago
>>>
>>> Tiago Lubiana
>>> Mestrando em Bioinformática, FCF/USP
>>> *Computational Systems Biology Laboratory (CSBL)*
>>> Telefone (laboratório): +55 (11) 2648-0240
>>> Telefone (pessoal): +55 (11) 954258000
>>>
>>>          [[alternative HTML version deleted]]
>>>
>>> _______________________________________________
>>> Bioc-devel using r-project.org mailing list
>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__stat.ethz.ch_mailman_listinfo_bioc-2Ddevel&d=DwIFaQ&c=eRAMFD45gAfqt84VtBcfhQ&r=BK7q3XeAvimeWdGbWY_wJYbW0WYiZvSXAJJKaaPhzWA&m=s7A_Q0fNE8fpu3Ob4M_wxwPfLdlY2QkGSbNwInWB2mo&s=b1C-benewoBq9gx9nDOXMGDReO7x4cpbfzQWEzUSRhc&e=
>>>
>>
> 
> 	[[alternative HTML version deleted]]
> 
> _______________________________________________
> Bioc-devel using r-project.org mailing list
> https://urldefense.proofpoint.com/v2/url?u=https-3A__stat.ethz.ch_mailman_listinfo_bioc-2Ddevel&d=DwIFaQ&c=eRAMFD45gAfqt84VtBcfhQ&r=BK7q3XeAvimeWdGbWY_wJYbW0WYiZvSXAJJKaaPhzWA&m=s7A_Q0fNE8fpu3Ob4M_wxwPfLdlY2QkGSbNwInWB2mo&s=b1C-benewoBq9gx9nDOXMGDReO7x4cpbfzQWEzUSRhc&e=
> 

-- 
Hervé Pagès

Program in Computational Biology
Division of Public Health Sciences
Fred Hutchinson Cancer Research Center
1100 Fairview Ave. N, M1-B514
P.O. Box 19024
Seattle, WA 98109-1024

E-mail: hpages using fredhutch.org
Phone:  (206) 667-5791
Fax:    (206) 667-1319