[Rd] Recent and upcoming changes to R-devel

[Duncan Murdoch]

On 11-07-06 3:46 AM, Stephan Wahlbrink wrote:
> Duncan Murdoch wrote [2011-07-05 17:21]:
>> On 05/07/2011 11:20 AM, Stephan Wahlbrink wrote:
>>> Dear developers,
>>>
>>> Duncan Murdoch wrote [2011-07-05 15:25]:
>>>> On 05/07/2011 6:52 AM, Tobias Verbeke wrote:
>>>>> L.S.
>>>>>
>>>>> On 07/05/2011 02:16 AM, Mark.Bravington at csiro.au wrote:
>>>>>> I may have misunderstood, but:
>>>>>>
>>>>>> Please could we have an optional installation that does not*not*
>>>>> byte-compile base and recommended?
>>>>>>
>>>>>> Reason: it's not possible to debug byte-compiled code-- at least not
>>>>> with the 'debug' package, which is quite widely used. I quite often
>>>>> end up using 'mtrace' on functions in base/recommended packages to
>>>>> figure out what they are doing. And sometimes I (and others)
>>>>> experiment with changing functions in base/recommended to improve
>>>>> functionality. That seems to be harder with BC versions, and might
>>>>> even be impossible, as best I can tell from hints in the documentation
>>>>> of 'compile').
>>>>>>
>>>>>> Personally, if I had to choose only one, I'd rather live with the
>>>>> speed penalty from not byte-compiling. But of course, if both are
>>>>> available, I could install both.
>>>>>
>>>>> I completely second this request. All speed improvements and the byte
>>>>> compiler in particular are leaps forward and I am very grateful and
>>>>> admiring towards the people that make this happen.
>>>>>
>>>>> That being said, 'moving away' from the sources (with the lazy loading
>>>>> files and byte-compilation) may be a step back for R package
>>> developers
>>>>> that (during development and maybe on separate development
>>> installations
>>>>> [as opposed to production installations of R]) require
>>>>> the sources of all packages to be efficient in their work.
>>>>>
>>>>> As many of you know there is an open source Eclipse/StatET visual
>>>>> debugger ready and for that application as well (similar to Mark's
>>>>> request) presence of non-compiled code is highly desirable.
>>>>>
>>>>> For the particular purpose of debugging R packages, I would even plead
>>>>> to go beyond the current options and support the addition of an
>>>>> R package install option that allows to include the sources (e.g. in
>>>>> a standard folder Rsrc/) in installed packages.
>>>>>
>>>>> I am fully aware that one can always fetch the source tarballs from
>>>>> CRAN for that purpose, but it would be much more easy if a simple
>>>>> installation option could put the R sources of a package in a separate
>>>>> folder [or archive inside an existing folder] such that R development
>>>>> tools (such as the Eclipse/StatET IDE) can offer inspection of sources
>>>>> or display them (e.g. during debugging) out of the box.
>>>>>
>>>>> If one has the srcref, one can always load the absolutely correct
>>> source
>>>>> code this way, even if one doesn't know the parent function with
>>>>> the source attribute.
>>>>>
>>>>> Any comments?
>>>>
>>>> I think these requests have already been met. If you modify the body of
>>>> a closure (as trace() does), then the byte compiled version is
>>>> discarded, and you go back to the regular interpreted code. If you
>>>> install packages with the R_KEEP_PKG_SOURCE=yes environment variable
>>>> set, the you keep all source for all functions. (It's attached to the
>>>> function itself, not as a file that may be out of date.) It's possible
>>>> that byte compiling turns off R_KEEP_PKG_SOURCE, but that is something
>>>> that is either easily fixed, or avoided by re-installing without byte
>>>> compiling.
>>>
>>> I don’t know how the new installation works exactly, but would it be
>>> possible, to simply install both types, the old expression bodies and
>>> the new byte-compiled, as single package at the same time?
>>
>> Yes, that's what is done.
>
> Perfect! And which version (byte-compiled or expressions) is used at
> runtime under which condition?

The byte code is executed, the interpreted version is displayed.  There 
are conditions under which the byte code is dropped.  I don't know a 
comprehensive list, but the idea is that if the body changes, then the 
compiled version is no longer valid.

Duncan Murdoch

>
> Thanks,
> Stephan
>
>
>>> This would
>>> allow the R user and developer to simply use the variant which is the
>>> best at the moment. If he wants to debug code, he can switch of the use
>>> of byte-compiled code and use the old R expressions (with attached
>>> srcrefs). If debugging is not required, he can profit from the
>>> byte-compiled version. The best would be a toggle, to switch it at
>>> runtime, but a startup option would be sufficient too.
>>>
>>> I think direct access to the code is one big advantage of open source
>>> software. For developer it makes it easier to find and fix bugs if
>>> something is wrong. But it can also help users a lot to understand how a
>>> function or algorithm works and learn from code written by other persons
>>> – if the access to the sources is easy.
>>>
>>> As long byte-code doesn’t support the debugging features of R, it is
>>> required for best debugging support to run the functions completely
>>> without byte-complied code. If I understood it correctly, byte-code
>>> frames would disable srcrefs as well as features like “step return” to
>>> that frames. Therefore I ask for a way that it is easy to switch between
>>> both execution types.
>>
>> What gave you that impression?
>>
>> Duncan Murdoch
>>
>>> Best,
>>> Stephan
>>>
>>>
>>>>
>>>> Duncan Murdoch
>>>>
>>>>> Best,
>>>>> Tobias
>>>>>
>>>>> P.S. One could even consider a post-install option e.g. to add 'real'
>>>>> R sources (and source references) to Windows packages (which are by
>>>>> definition already 'installed' and for which such information is not
>>>>> by default included in the CRAN binaries of these packages).
>>>>>
>>>>>>>> Prof Brian Ripley wrote:
>>>>>>>> There was an R-core meeting the week before last, and various
>>>>> planned
>>>>>>>> changes will appear in R-devel over the next few weeks.
>>>>>>>>
>>>>>>>> These are changes planned for R 2.14.0 scheduled for Oct 31.
>>> As we
>>>>>>>> are sick of people referring to R-devel as '2.14' or '2.14.0',
>>> that
>>>>>>>> version number will not be used until we reach 2.14.0 alpha. You
>>>>>>>> will be able to have a package depend on an svn version number
>>> when
>>>>>>>> referring to R-devel rather than using R (>= 2.14.0).
>>>>>>>>
>>>>>>>> All packages are installed with lazy-loading (there were 72 CRAN
>>>>>>>> packages and 8 BioC packages which opted out). This means that
>>> the
>>>>>>>> code is always parsed at install time which inter alia simplifies
>>>>> the
>>>>>>>> descriptions. R 2.13.1 RC warns on installation about packages
>>> which
>>>>>>>> ask not to be lazy-loaded, and R-devel ignores such requests
>>> (with a
>>>>>>>> warning).
>>>>>>>>
>>>>>>>> In the near future all packages will have a name space. If the
>>>>>>>> sources do not contain one, a default NAMESPACE file will be
>>> added.
>>>>>>>> This again will simplify the descriptions and also a lot of
>>> internal
>>>>>>>> code. Maintainers of packages without name spaces (currently
>>> 42% of
>>>>>>>> CRAN) are encouraged to add one themselves.
>>>>>>>>
>>>>>>>> R-devel is installed with the base and recommended packages
>>>>>>>> byte-compiled (the equivalent of 'make bytecode' in R 2.13.x, but
>>>>>>>> done less inefficiently). There is a new option R CMD INSTALL
>>>>>>>> --byte-compile to byte-compile contributed packages, but that
>>>>> remains
>>>>>>>> optional.
>>>>>>>> Byte-compilation is quite expensive (so you definitely want to
>>> do it
>>>>>>>> at install time, which requires lazy-loading), and relatively few
>>>>>>>> packages benefit appreciably from byte-compilation. A larger
>>> number
>>>>>>>> of packages benefit from byte-compilation of R itself: for
>>> example
>>>>>>>> AER runs its checks 10% faster. The byte-compiler technology is
>>>>>>>> thanks to Luke Tierney.
>>>>>>>>
>>>>>>>> There is support for figures in Rd files: currently with a
>>>>> first-pass
>>>>>>>> implementation (thanks to Duncan Murdoch).
>>>
>>> --
>>> Stephan Wahlbrink
>>> Humboldtstr. 19
>>> 44137 Dortmund
>>> Germany
>>> http://www.walware.de/goto/opensource
>>>
>>
>>