[Bioc-devel] vignette problems
campos
campos at mpipz.mpg.de
Thu Apr 5 13:45:57 CEST 2018
Hey Martin,
I pushed new changes since last friday but in
https://bioconductor.org/checkResults/3.7/bioc-LATEST/STAN/ says that
the last change date was friday. Any idea what is the problem?
I have tried to fix the problems with memory and all you told me.
Best,
Rafael
On 03.04.2018 17:06, Martin Morgan wrote:
> Please use 'reply all' so that the mailing list remains engaged.
>
> Check out the release schedule
>
> http://bioconductor.org/developers/release-schedule/
>
> in particular
>
> Wednesday April 25
>
> - Deadline for packages passing ‘‘R CMD build’’ and ‘‘R CMD check’’
> without errors or warnings.
>
> so you still have time to get your package in order.
>
> Using the same techniques as before, I still see valgrind problems,
> the first being
>
> > hmm_fitted_poilog = fitHMM(trainRegions, hmm_poilog,
> sizeFactors=sizeFactors, maxIters=10)
> ==24916== Invalid write of size 4
> ==24916== at 0x4BA93FD7:
> TransitionMatrix::updateAuxiliaries(double**, double***, double*,
> int*, int, int**, int, double, int) (TransitionMatrix.cpp:292)
> ==24916== by 0x4BA77934: HMM::updateSampleAux(double***, int*, int,
> double**, double**, double**, double***, double*, int*, int*, int*,
> int**, double***, SEXPREC*, SEXPREC*, int, double, int, int, int)
> (HMM.cpp:771)
> ==24916== by 0x4BA7896D: HMM::BaumWelch[abi:cxx11](double***, int*,
> int, int, int**, int*, int*, int*, int, int, int**, double***,
> SEXPREC*, SEXPREC*, int, double, double, int, int) (HMM.cpp:1076)
> ==24916== by 0x4BA8FEFD: RHMMFit (RWrapper.cpp:1494)
> ==24916== by 0x4F2992D: R_doDotCall (dotcode.c:692)
> ==24916== by 0x4F339D5: do_dotcall (dotcode.c:1252)
> ==24916== by 0x4F81BA6: bcEval (eval.c:6771)
> ==24916== by 0x4F6E963: Rf_eval (eval.c:624)
> ==24916== by 0x4F71188: R_execClosure (eval.c:1764)
> ==24916== by 0x4F70E7C: Rf_applyClosure (eval.c:1692)
> ==24916== by 0x4F6F18B: Rf_eval (eval.c:747)
> ==24916== by 0x4F74B12: do_set (eval.c:2774)
> ==24916== Address 0x2e73a294 is 4 bytes inside a block of size 5 alloc'd
> ==24916== at 0x4C2DB8F: malloc (in
> /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
> ==24916== by 0x4BA93FA6:
> TransitionMatrix::updateAuxiliaries(double**, double***, double*,
> int*, int, int**, int, double, int) (TransitionMatrix.cpp:289)
> ==24916== by 0x4BA77934: HMM::updateSampleAux(double***, int*, int,
> double**, double**, double**, double***, double*, int*, int*, int*,
> int**, double***, SEXPREC*, SEXPREC*, int, double, int, int, int)
> (HMM.cpp:771)
> ==24916== by 0x4BA7896D: HMM::BaumWelch[abi:cxx11](double***, int*,
> int, int, int**, int*, int*, int*, int, int, int**, double***,
> SEXPREC*, SEXPREC*, int, double, double, int, int) (HMM.cpp:1076)
> ==24916== by 0x4BA8FEFD: RHMMFit (RWrapper.cpp:1494)
> ==24916== by 0x4F2992D: R_doDotCall (dotcode.c:692)
> ==24916== by 0x4F339D5: do_dotcall (dotcode.c:1252)
> ==24916== by 0x4F81BA6: bcEval (eval.c:6771)
> ==24916== by 0x4F6E963: Rf_eval (eval.c:624)
> ==24916== by 0x4F71188: R_execClosure (eval.c:1764)
> ==24916== by 0x4F70E7C: Rf_applyClosure (eval.c:1692)
> ==24916== by 0x4F6F18B: Rf_eval (eval.c:747)
> ==24916==
>
> This seems to be the exact same code as in the problem that you fixed
> at another location. Actually, I would guess that all of these
>
> grep --color -nH -e ")\*ncores+1" *
> HMM.cpp:784: int *myStateBuckets = (int*)malloc(sizeof(int)*ncores+1);
> MultivariateGaussian.cpp:295: int *myDimBuckets =
> (int*)malloc(sizeof(int)*ncores+1);
> MultivariateGaussian.cpp:475: int *myDimBuckets =
> (int*)malloc(sizeof(int)*ncores+1);
> TransitionMatrix.cpp:132: int *myStateBuckets =
> (int*)malloc(sizeof(int)*ncores+1);
> TransitionMatrix.cpp:289: int *myStateBuckets =
> (int*)malloc(sizeof(int)*ncores+1);
>
> are the same problem. Also, usually code that has been copy/pasted
> like this can instead be refactored to a single function call, so a
> bug can be fixed in one place.
>
> I still see a number of compiler warnings, the first of which is
>
> STAN master$ R CMD INSTALL .
> Bioconductor version 3.7 (BiocInstaller 1.29.5), ?biocLite for help
> * installing to library
> '/home/mtmorgan/R/x86_64-pc-linux-gnu-library/3.5-Bioc-3.7'
> * installing *source* package 'STAN' ...
> ** libs
> g++ -I"/home/mtmorgan/bin/R-3-5-branch/include" -DNDEBUG
> -I/usr/local/include -D_RDLL_ -fopenmp -fpic -g -O0 -Wall -pedantic
> -c HMM.cpp -o HMM.o
> HMM.cpp: In member function ‘virtual void
> HMM::calcEmissionProbs(double***, double**, int*, int, int**, int*,
> int*, int**, int, int, int*)’:
> HMM.cpp:112:15: warning: unused variable ‘j’ [-Wunused-variable]
> int i,j,t,k;
> ^
> It really pays to clean these up; most are harmless, but they obscure
> the more important warnings.
>
> Martin
>
>
> On 04/03/2018 09:58 AM, campos wrote:
>> Hi Martin,
>>
>> when I run now valgrind seems that there are no problems. I pushed
>> yesterday the changes, was I still on time for the new release? We
>> want to publish the changes and it would be really helpful if the
>> package is running on Bioconductor.
>>
>> Thank you very much,
>>
>> Rafael
>>
>>
>> On 02.04.2018 02:51, Martin Morgan wrote:
>>>
>>>
>>> On 04/01/2018 08:06 PM, Martin Morgan wrote:
>>>>
>>>>
>>>> On 04/01/2018 03:53 PM, campos wrote:
>>>>> Dear Martin,
>>>>>
>>>>> I am trying to fix this problem but I am really lost... Do you
>>>>> mean C++ code? Becasue there is no C code in the whole package. I
>>>>> really don't know what the problem might be.
>>>>
>>>> Please keep the conversation on the bioc-devel mailing list, so
>>>> that others can learn or help.
>>>>
>>>> I use Linux, not Mac, but C (which I used to mean your C++ code)
>>>> errors often occur on all platforms but are only visible as a
>>>> segfault on one. I created the vignette R code with
>>>>
>>>> cd vignettes
>>>> R CMD Stangle STAN-knitr.Rmd
>>>>
>>>> This produces a file STAN-knitr.R. I then ran your R code with
>>>> valgrind
>>>>
>>>> R -d valgrind -f STAN-knitr.R
>>>>
>>>> this runs much slower than without valgrind. The first error
>>>> reported by valgrind was
>>>>
>>>>
>>>> > ##
>>>> ----STAN-PoiLog-----------------------------------------------------------
>>>>
>>>> > nStates = 10
>>>> > hmm_poilog = initHMM(trainRegions, nStates, "PoissonLogNormal",
>>>> sizeFactors)
>>>> > hmm_fitted_poilog = fitHMM(trainRegions, hmm_poilog,
>>>> sizeFactors=sizeFactors, maxIters=10)
>>>> [1] 6
>>>> ==22304== Invalid write of size 4
>>>> ==22304== at 0x4B489316: HMM::BaumWelch[abi:cxx11](double***,
>>>> int*, int, int, int**, int*, int*, int*, int, int, int**,
>>>> double***, SEXPREC*, SEXPREC*, int, double, double, int, int)
>>>> (HMM.cpp:998)
>>>> ==22304== by 0x4B4A0EFF: RHMMFit (RWrapper.cpp:1494)
>>>> ==22304== by 0x4F2992D: R_doDotCall (dotcode.c:692)
>>>> ==22304== by 0x4F339D5: do_dotcall (dotcode.c:1252)
>>>> ==22304== by 0x4F81BA6: bcEval (eval.c:6771)
>>>> ==22304== by 0x4F6E963: Rf_eval (eval.c:624)
>>>> ==22304== by 0x4F71188: R_execClosure (eval.c:1764)
>>>> ==22304== by 0x4F70E7C: Rf_applyClosure (eval.c:1692)
>>>> ==22304== by 0x4F6F18B: Rf_eval (eval.c:747)
>>>> ==22304== by 0x4F74B12: do_set (eval.c:2774)
>>>> ==22304== by 0x4F6EDF5: Rf_eval (eval.c:699)
>>>> ==22304== by 0x4FB7BEE: Rf_ReplIteration (main.c:258)
>>>> ==22304== Address 0x238b28f4 is 4 bytes inside a block of size 5
>>>> alloc'd
>>>> ==22304== at 0x4C2DB8F: malloc (in
>>>> /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
>>>> ==22304== by 0x4B4892E5: HMM::BaumWelch[abi:cxx11](double***,
>>>> int*, int, int, int**, int*, int*, int*, int, int, int**,
>>>> double***, SEXPREC*, SEXPREC*, int, double, double, int, int)
>>>> (HMM.cpp:995)
>>>> ==22304== by 0x4B4A0EFF: RHMMFit (RWrapper.cpp:1494)
>>>> ==22304== by 0x4F2992D: R_doDotCall (dotcode.c:692)
>>>> ==22304== by 0x4F339D5: do_dotcall (dotcode.c:1252)
>>>> ==22304== by 0x4F81BA6: bcEval (eval.c:6771)
>>>> ==22304== by 0x4F6E963: Rf_eval (eval.c:624)
>>>> ==22304== by 0x4F71188: R_execClosure (eval.c:1764)
>>>> ==22304== by 0x4F70E7C: Rf_applyClosure (eval.c:1692)
>>>> ==22304== by 0x4F6F18B: Rf_eval (eval.c:747)
>>>> ==22304== by 0x4F74B12: do_set (eval.c:2774)
>>>> ==22304== by 0x4F6EDF5: Rf_eval (eval.c:699)
>>>>
>>>> 'Invalid write' suggests that you are writing after the end of
>>>> memory that you'd allocated. I looked at the C code at the line
>>>> where the error occurs as indicated in the stack trace, HMM.cpp:998
>>>> which is the assigment myStateBucks[i] = 0 in the loop
>>>>
>>>> int *myStateBuckets = (int*)malloc(sizeof(int)*ncores+1);
>>>> for(i=0; i<=ncores; i++)
>>>> {
>>>> myStateBuckets[i] = 0;
>>>> }
>>>>
>>>> The argument to malloc (where he memory was allocated, at line 995)
>>>> should be the number of bytes to allocate and it should have been
>>>> memory for ncores + 1 'int'
>>>>
>>>> malloc(sizeof(int) * (ncores + 1))
>>>>
>>>> rather than what you wrote, which is memory for ncores ints plus 1
>>>> byte.
>>>>
>>>> C++ code would avoid the need for such explicit memory management,
>>>> e.g., using a vector from the standard template library
>>>>
>>>> std::vector<int> myStateBuckets(ncores);
>>>
>>> oops, std::vector<int> myStateBuckets(ncores + 1); !
>>>>
>>>> There were may other valgrind errors, but I do not know whether
>>>> these are from similar programming errors, or a consequence of this
>>>> one.
>>>>
>>>> Martin
>>>>
>>>>> Thanks,
>>>>> Rafael
>>>>>>
>>>>>> On 03/29/2018 01:07 PM, campos wrote:
>>>>>>> Dear bioc-devel team,
>>>>>>>
>>>>>>> I have made some changes in the package STAN and although it
>>>>>>> seems to install correctly, I have problems with timeout and
>>>>>>> error in windows... Could someone help me to improve the time?
>>>>>>>
>>>>>>> https://bioconductor.org/checkResults/3.7/bioc-LATEST/STAN/
>>>>>>
>>>>>> it looks like, with your most recent commit (at the top of the
>>>>>> page, 'Snapshot Date', 'Last Commit', 'Last Changed Date'), the
>>>>>> package built on Windows and Linux.
>>>>>>
>>>>>> There is a segfault on Mac, which is likely a programming error
>>>>>> in your C code. It could be debugged perhaps using valgrind or
>>>>>> similar tools, but the first step would be to isolate the code to
>>>>>> something more easily reproduced than the full vignette. It would
>>>>>> also help to clean up the C code so that it compiles without
>>>>>> warnings with the -Wall -pedantic flags
>>>>>>
>>>>>> Martin
>>>>>>
>>>>>>>
>>>>>>> Best,
>>>>>>>
>>>>>>> Rafael
>>>>>>>
>>>>>>>
>>>>>>> On 28.03.2018 01:08, Martin Morgan wrote:
>>>>>>>> When I try and install the version on the master branch of the
>>>>>>>> Bioconductor git repository I get
>>>>>>>>
>>>>>>>> STAN master$ Rdev --vanilla CMD INSTALL .
>>>>>>>> * installing to library
>>>>>>>> ‘/home/mtmorgan/R/x86_64-pc-linux-gnu-library
>>>>>>>> ...
>>>>>>>> ** testing if installed package can be loaded
>>>>>>>> Error: package or namespace load failed for 'STAN' in
>>>>>>>> namespaceExport(ns, exports):
>>>>>>>> undefined exports: viterbi2Gviz
>>>>>>>>
>>>>>>>>
>>>>>>>> This comes about in a rather interesting way because the body
>>>>>>>> of plotViterbi is not defined
>>>>>>>>
>>>>>>>> plotViterbi <- function(viterbi, regions, gen, chrom, from, to,
>>>>>>>> statecols, col)
>>>>>>>>
>>>>>>>> #'
>>>>>>>> ...
>>>>>>>>
>>>>>>>> Can you please commit a version of the package that installs?
>>>>>>>>
>>>>>>>> Martin
>>>>>>>>
>>>>>>>> On 03/27/2018 06:42 PM, campos wrote:
>>>>>>>>> Dear bioc-devel team,
>>>>>>>>>
>>>>>>>>> I am developing the STAN packages and I am running into
>>>>>>>>> problems when trying to build my package. The problem is the
>>>>>>>>> following:
>>>>>>>>>
>>>>>>>>> Error in vignette_type(Outfile) :
>>>>>>>>> Vignette product 'STAN.tex' does not have a known filename
>>>>>>>>> extension ('NA')
>>>>>>>>> ERROR: installing vignettes failed
>>>>>>>>> * removing '/tmp/Rtmp925Iru/Rinst63471ff1efdc/STAN'
>>>>>>>>> I tried to build the package in old versions (which they used
>>>>>>>>> to work) and I run in other problems but in this case is:
>>>>>>>>>
>>>>>>>>> Error in texi2dvi(file = file, pdf = TRUE, clean = clean,
>>>>>>>>> quiet = quiet, :
>>>>>>>>> Running 'texi2dvi' on 'STAN.tex' failed.
>>>>>>>>> LaTeX errors:
>>>>>>>>> ! LaTeX Error: File `beramono.sty' not found.
>>>>>>>>>
>>>>>>>>> Type X to quit or <RETURN> to proceed,
>>>>>>>>> or enter new name. (Default extension: sty)
>>>>>>>>>
>>>>>>>>> Could you help me with this problem?
>>>>>>>>>
>>>>>>>>> Thank you very much,
>>>>>>>>>
>>>>>>>>> Rafael
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Bioc-devel at r-project.org mailing list
>>>>>>>>> https://stat.ethz.ch/mailman/listinfo/bioc-devel
>>>>>>>>
>>>>>>>>
>>>>>>>> This email message may contain legally privileged and/or
>>>>>>>> confidential information. If you are not the intended
>>>>>>>> recipient(s), or the employee or agent responsible for the
>>>>>>>> delivery of this message to the intended recipient(s), you are
>>>>>>>> hereby notified that any disclosure, copying, distribution, or
>>>>>>>> use of this email message is prohibited. If you have received
>>>>>>>> this message in error, please notify the sender immediately by
>>>>>>>> e-mail and delete this email message from your computer. Thank
>>>>>>>> you.
>>>>>>>
>>>>>>
>>>>>>
>>>>>> This email message may contain legally privileged and/or
>>>>>> confidential information. If you are not the intended
>>>>>> recipient(s), or the employee or agent responsible for the
>>>>>> delivery of this message to the intended recipient(s), you are
>>>>>> hereby notified that any disclosure, copying, distribution, or
>>>>>> use of this email message is prohibited. If you have received
>>>>>> this message in error, please notify the sender immediately by
>>>>>> e-mail and delete this email message from your computer. Thank you.
>>>>>
>>>>
>>>>
>>>> This email message may contain legally privileged
>>>> and/or...{{dropped:2}}
>>>>
>>>> _______________________________________________
>>>> Bioc-devel at r-project.org mailing list
>>>> https://stat.ethz.ch/mailman/listinfo/bioc-devel
>>>
>>>
>>> This email message may contain legally privileged and/or
>>> confidential information. If you are not the intended recipient(s),
>>> or the employee or agent responsible for the delivery of this
>>> message to the intended recipient(s), you are hereby notified that
>>> any disclosure, copying, distribution, or use of this email message
>>> is prohibited. If you have received this message in error, please
>>> notify the sender immediately by e-mail and delete this email
>>> message from your computer. Thank you.
>>
>
>
> This email message may contain legally privileged and/or confidential
> information. If you are not the intended recipient(s), or the
> employee or agent responsible for the delivery of this message to the
> intended recipient(s), you are hereby notified that any disclosure,
> copying, distribution, or use of this email message is prohibited. If
> you have received this message in error, please notify the sender
> immediately by e-mail and delete this email message from your
> computer. Thank you.
More information about the Bioc-devel
mailing list