[R-pkg-devel] ERROR building MixAll on Windows platform

[Tomas Kalibera]

On 1/19/24 15:18, Serge wrote:
> This post is a continuation of the post *[R-pkg-devel] Does 
> dependencies up to date on the pretest CRAN infrastructure*
>
> I made more (unsuccessful) tries:
>
> - I installed a Windows 11 version in a VM on my compuiter and try to 
> buid the MixAll package using Rtools42 and Rtools43 (it's quite easy, 
> and funny, to do it on windows: you have just to rename C:\rtools42 as 
> C:\rtools43).
There should be no renaming, instead, one should use R 4.2.x with 
Rtools42 and R 4.3.x (or current R-devel) with Rtools43. All of these 
can co-exist (be installed at the same time). Mixing the two could lead 
to different failures. I understand you want to test different versions 
of GCC, but to do that reliable you would have to rebuild the rest of 
Rtools with that, or just the part that is needed by the package.
>
> The result is that MixAll is build using the 4.2 version and the buid 
> failed using the 4.3 version.
Please make sure to use Rtools43 (the real one) with R 4.3.
>
> - I installed the version 12.3 of gcc on ubuntu (the same version used 
> on windows) and could build the package without problem
>
> - Inspecting the log of the Rtools4.3 
> (https://cran.r-project.org/bin/windows/Rtools/rtools43/news.html) and 
> g++12..3 (https://gcc.gnu.org/gcc-12/changes.html) does not give insight.
>
> - The package is dependent of the rtkore package which use extensive 
> use of templated class. As rtkore is a port for R of stk++, I try to 
> compile the stk++ library on windows (using g++12). All tests are 
> compiled without any troubles.
>
> These attempts eliminate some causes, but don't give me any insight 
> why MixAll (and blockcluster) failed to be build on the Windows-devel 
> platform. It seems related to Rtools43. Does anyone else (using for 
> exemple Rcppeigen) is experiencing this problem ?

If this is GCC running out of memory on Windows but not Linux, when 
there really should be enough of memory available (i.e. due to the 
problem that Uwe described, maybe the internal GC in GCC is asking for 
too much memory for its heap given that the OS is not overcomiting), you 
can try narrowing it down to a standalone C++ program (independent on R, 
Rtools, R packages, compilable on both Windows and Linux with disabled 
optimizations, etc) that still exhibits the problem and then submitting 
it in a bug report to GCC bugzilla.

In such case, it could be that some heuristics in the collector could be 
improved.

If you have such standalone example, it would be easier to test 
different versions of GCC or even bisect to a concrete GCC version. You 
might also compare memory usage when compiling and cross-compiling.

You might also be able to find a work-around for your package by 
disabling some compiler optimizations. Also if you can narrow this down 
to a concrete GCC optimization then it would help to mention that in the 
bug report.

Certainly all of this would be out of scope of R-(pkg)-devel, this would 
rather be a GCC/Windows thing.

Tomas

>
> Serge
>
>
>
> Le 14/01/2024 à 18:50, Uwe Ligges a écrit :
>>
>>
>> On 13.01.2024 15:01, Uwe Ligges wrote:
>>> Fascinating, now it worked with the latest winbuilder submission 3 
>>> times in a row when I checked it manually. So maybe Ivan was right 
>>> and there was a very demanding set of other packages compiling at 
>>> the same time?
>>> I don't know.
>>>
>>> Serge, Can you somply submit your latest winbuilder upload to CRAN?
>>
>> Really, I inspected some more. The underlying issue is simple:
>> The C++ compiler used under Windows asks for precomitted memory. If 
>> several processes are running at the same time, a lot of memory is 
>> precomitted. And Windows does not use it for other processes, even if 
>> almost nothing is actually used.
>> So while the used memory may be around 50GB, all of the rest (of 756 
>> GB including swap space) may have been precomitted (but unused) and 
>> new processes failed to start correctly. Grrrr.
>>
>> Best,
>> Uwe Ligges
>>
>>
>>
>>
>>
>>> Best,
>>> Uwe Ligges
>>>
>>>
>>>
>>> On 13.01.2024 14:12, Uwe Ligges wrote:
>>>> I can take a look, but not sure if I get to it before monday.
>>>> I haven't seen it for any other packages recently.
>>>>
>>>> My suspicion is currently a strange mix of cmd.exe and sh.exe 
>>>> calls. But this is a very wild guess.
>>>>
>>>> Best,
>>>> Uwe
>>>>
>>>> On 13.01.2024 14:08, Uwe Ligges wrote:
>>>>>
>>>>>
>>>>> On 13.01.2024 10:10, Ivan Krylov via R-package-devel wrote:
>>>>>> В Fri, 12 Jan 2024 21:19:00 +0100
>>>>>> Serge <Serge.Iovleff using stkpp.org> пишет:
>>>>>>
>>>>>>> After somme minor midficiations, I make a try on the winbuilder 
>>>>>>> site.
>>>>>>> I was able to build the archive with the static library
>>>>>>> but I get again a Bad address error. You can have a look to
>>>>>>>
>>>>>>> https://win-builder.r-project.org/bw47qsMX3HTd/00install.out
>>>>>>
>>>>>> I think that Win-Builder is running out of memory. It took some
>>>>>> experimenting, but I was able to reproduce something like this using
>>>>>> the following:
>>>>>>
>>>>>> 1. Set the swap file in the Windows settings to minimal recommended
>>>>>> size and disable its automatic growth
>>>>>>
>>>>>> 2. Write and run a program that does malloc(LARGE_NUMBER); 
>>>>>> getchar();
>>>>>> so that almost all physical memory is allocated
>>>>>>
>>>>>> 3. Run gcc -DFOO=`/path/to/Rscript -e 'some script'` & many times
>>>>>>
>>>>>> I got a lot of interesting errors, including the "Bad address":
>>>>>>
>>>>>> Warnings:
>>>>>> 1: .getGeneric(f, , package) : internal error -4 in R_decompress1
>>>>>> 2: package "methods" in options("defaultPackages") was not found
>>>>>>
>>>>>> 0 [main] bash (2892) child_copy: cygheap read copy failed,
>>>>>> 0x0..0x800025420, done 0, windows pid 2892, Win32 error 299
>>>>>>
>>>>>> 0 [main] bash (3256) C:\rtools43\usr\bin\bash.exe: *** fatal 
>>>>>> error in
>>>>>> forked process - MEM_COMMIT failed, Win32 error 1455
>>>>>>
>>>>>> -bash: fork: retry: Resource temporarily unavailable
>>>>>>
>>>>>> -bash: R-devel/bin/Rscript.exe: Bad address
>>>>>
>>>>> The above indeed happens if not sufficient memory would be available.
>>>>> Important to know: This includes unused but committed memory which 
>>>>> may be a lot.
>>>>> But I doubt it is the case on winbuilder as the machines has 256GB 
>>>>> or more (depending in the machine) and additionally 500GB swap 
>>>>> space on SSD.
>>>>>
>>>>> Best,
>>>>> Uwe
>>>>>
>>>>>
>>>>>> Your package is written in C++, but that by itself shouldn't 
>>>>>> disqualify
>>>>>> it. On my Linux computer, /usr/bin/time R -e
>>>>>> 'install.packages("MixAll")' says that the installation takes 
>>>>>> slightly
>>>>>> less than a gigabyte of memory ("912516maxresident k"), which is par
>>>>>> the course for such packages. (My small Rcpp-using package takes
>>>>>> approximately half a gigabyte by the same metric.)
>>>>>>
>>>>>> I'm still not 100% sure (if Win-Builder is running out of memory, 
>>>>>> why
>>>>>> are you seeing "Bad address" only and not the rest of the carnage?),
>>>>>> but I'm not seeing a problem with your package, either. If EFAULT is
>>>>>> Cygwin's way of saying "I caught a bad pointer in your system call"
>>>>>> (which, I must stress, is happening inside /bin/sh, not your package
>>>>>> or even R at all), it's not impossible that Win-Builder is having
>>>>>> hardware problems. Unfortunately, they take a lot of effort and
>>>>>> downtime to diagnose and could be hiding anywhere from RAM to the 
>>>>>> power
>>>>>> supply.
>>>>>>
>>>>>
>>>>> ______________________________________________
>>>>> R-package-devel using r-project.org mailing list
>>>>> https://stat.ethz.ch/mailman/listinfo/r-package-devel
>>>>
>>>> ______________________________________________
>>>> R-package-devel using r-project.org mailing list
>>>> https://stat.ethz.ch/mailman/listinfo/r-package-devel
>>>
>>> ______________________________________________
>>> R-package-devel using r-project.org mailing list
>>> https://stat.ethz.ch/mailman/listinfo/r-package-devel
>
> ______________________________________________
> R-package-devel using r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-package-devel