[R-sig-Debian] update.packages() as ordinary user, /usr/lib/R/site-library is not writable

[Chris Evans]

Oh dear.  The saga continues.  See below.

jranke at uni-bremen.de sent the following  at 22/09/11 12:52:
> Zitat von Chris Evans <chrishold at psyctc.org>:
>
>> Huge thanks to Johannes (sorry, Johannes, I think I implied that Dirk 
>> was doing all the hard work in rolling things in my earlier reply, 
>> the "you" is plural!)
>
> Never mind - you are absolutely right that Dirk is doing the heavy 
> lifting, i.e. packaging and I just recompile some of his packages for 
> Debian stable.
>
>> What I've done:
>> A) removed r-cran-rgl and r-cran-xml using synaptic as root
>> B) re-entered new R session as user chris, confirmed that the 
>> packages had gone using installed.packages() and grep()!
>> C) ran "adduser chris staff" as root
>> D) logged off gnome session (ah, that bit's a pain when making group 
>> membership changes which won't otherwise propagate to current session!)
>> E) logged back in!
>> F) checked I was now member of staff: yes!
>> G) checked I can write to /usr/local/lib/R/site-library: yes!
>> H) ran R as user chris ... now it gets interesting and frustrating:
>>     no damn joy, update.packages and selecting only abind to update 
>> without specifying instlib says it runs fine but next time I run 
>> update.packages it still shows the old version and wants to do the 
>> update again so I
> Where does abind live on your system? update.packages() always 
> mentions where it finds its candidates.
>
>> I) (and this was a debugging mistake for which I apologise) did the 
>> same but specifying instlib as /usr/local ... and it runs fine again 
>> (this time I checked that it had updated in /usr/local/... and there 
>> it was in solitary splendour as my first updated library/package 
>> there) ... but  ...
>> J) that hasn't gained me anything (I suspect the previous update 
>> without the "instlib=" may have been fine too) as update.packages 
>> still wants to update it ...
>> K) so I get thoughtful: clearly update.packages is reading from 
>> /usr/lib not /usr/local ...
>> L) so I check .libPaths():
>>> .libPaths()
>> [1] "/home/chris/R/i486-pc-linux-gnu-library/2.13"
>
> So at some point in time you let update.packages() create a directory 
> in your home directory that would have precedence over system wide 
> installations.

Yes.  I think the first time I ran the update.packages() it asked me if 
I'd like to create that directory and I said yes.

>
>> [2] "/usr/local/lib/R/site-library"
>> [3] "/usr/lib/R/site-library"
>> [4] "/usr/lib/R/library"
>> M) Hm.  Looks good to me but I see Dirk is telling me /usr/local/... 
>> should be first so I go to /etc/R/Renviron and I find:
>> R_LIBS_SITE=${R_LIBS_SITE-'/usr/local/lib/R/site-library:/usr/lib/R/site-library:/usr/lib/R/library'} 
>>
>> and nothing I can see in Renviron.site that would override that.
>>
>> I _THINK_ I see a logic:  installation is happening "first come, 
>> first used" on the libPaths but update.packages() is reading from all 
>> the library locations and the later are overriting the earlier ... 
>> that would explain things but surely it would mean that no-one, 
>> including Dirk, would be able to run update.packages() successfully 
>> as a non-root user.
>
> Yes, you and I can run update.packages() successfully, but only on 
> paths with write permissions. As it does not seem a good idea to give 
> a normal user write permissions on /usr/lib/ anywhere, I would suggest 
> doing (from within R)
>
> update.packages(.libPaths()[1])
> update.packages(.libPaths()[2])
>
> This has just worked nicely for me on my Debian stable laptop.

I think on my machine the equivalent is update.packages(.libPaths()[2]) 
and then [3].  I did run [1] and [2] and the first run prompted me for 
the CRAN repository and then exited silently and instantly: I assume 
there's no package in my /home/... directory that needs updating.  Then 
I ran for .libPaths()[2] and it no longer needed to prompt for 
repository and exited silently and instantly.  Then I ran for 
.libPaths()[3] and it asked me about many libraries that could be 
updated starting with abind telling me it was in /usr/lib/R/site-library 
and at version 1.1.0 with 1.3.0 available via CRAN but when I asked it 
to upgrade abind it failed complaining that it couldn't write to the 
directory.  So what I did then was:

update.packages(.libPaths()[3],instlib="/usr/local/lib/R/site-lib")

That seemed to run fine, installed to /usr/local/... and I confess I've 
now gone into /usr/lib/R/... and deleted the now outdated libraries 
there and then used mv to move the remaining ones to /usr/local/...

All that begs the question how I got myself here.  I think the learning 
point is that there may be a way to make the instructions in the deb 
packages in stable (well, in the next stable I guess) spell out the 
relationship between the deb repository in stable, the deb repository in 
CRAN (with a link to the page in CRAN about the debs there) and the 
non-deb libraries in CRAN.  You might spell out options which I have as:
1) you can stay with the debs in stable, and update within R as an 
ordinary user, you won't get automatic R updates but you should get the 
package updates that are possible for that version of R
2) you can go with the debs in CRAN which will usually give you the 
latest version of R shortly after it is released.  In that  case [just 
my take and I can see that I could be quite wrong] you are best off 
doing  a complete remove of the version of R you installed from your 
Debian stable mirror using synaptic (or your own preferred package 
handler), _before_ adding the necessary line in /etc/apt/sources.list to 
get your local CRAN deb repository, after which you can do a clean 
install of R
3) you are probably wise to do adduser yourself staff to get write 
access to /usr/local/lib/R/site-library/
4) you should then find that things update fine through R and 
update.packages() running R as an ordinary user and you will be getting 
the latest R very soon after each upgrade and the latest upgrades of the 
packages you have installed.

It sounds to me as if there's another option which is use the debian 
backports deb repository/mirror but I'm not sure I understand the 
relationship between that option and the option of going to the CRAN debs.

Very best all & thanks again to you both Dirk & Johannes,

Chris


... rest of Johannes's reply and the thread follows for the record ...

>
> The other packages should (please correct me if I am wrong Dirk) 
> remain under control of the Debian package management.
>
> If you want to update such packages as well, you need to bug me or 
> help me in creating those backports (or use apt pinning e.g. to 
> testing, but I think this would make things too complicated). For 
> example, I only update R base packages on CRAN/Debian (like 
> r-cran-foreign, r-cran-nlme etc.) when a new R version is released. 
> And there are a lot of packages that Dirk meticulously updates in 
> Debian unstable that I have never backported for Debian stable, like 
> r-cran-xml and r-cran-rgl.
>
> Regards,
>
> Johannes
>
> P.S.: I also mainly use my own R distribution for Debian stable on 
> servers, just recently started to use it on my laptop (going back from 
> Ubuntu).
>
>
>
>>
>> So now I'm baffled as to how to be friendly with R and linux/Debian 
>> on this machine!
>>
>> Perhaps I should say that I would like to work by:
>> 1) having the cran deb line in /etc/apt/sources.list and doing basic 
>> installation of R from there as my understanding is that that will 
>> get me the latest R for Debian when you (plural) have rolled it and 
>> put it on CRAN getting me round the downloading and replacing I used 
>> to do on windoze and also making it pretty likely that you'll have 
>> sorted out dependency problems etc.
>> 2) would like then to rely on update.packages() from within R
>> 3) but running as local user as I can see the logic of that
>>
>> #1 and #2 are what I've always done on my Debian web server (where I 
>> was running some cgi-bin scripts invoking R) but I always did the 
>> update.packages() as root.  I've now moved that server to a shell 
>> account at mythic-beasts (great guys) so I'll have to do #3 for that 
>> too in future and though I could update as root on this laptop I do 
>> see the logic of not doing that and clearly you both and presumably 
>> many others on this list do #1, #2 and #3 so why is it not working 
>> for me?!
>>
>> Suggestions gratefully received!
>>
>> Very best & karma to all,
>>
>> Chris
>>
>>
>>
>> Dirk Eddelbuettel sent the following  at 21/09/11 13:49:
>>> Hi Chris,
>>>
>>> Thanks for patient follow-up even after I (as Johannes noted) evidently
>>> overlooked key parts of your first email.
>>>
>>> On 21 September 2011 at 07:49, Chris Evans wrote:
>>> | I ran synaptic from the Debian application menu which defaults to 
>>> root
>>> | after askling for the root password the first time I invoke it 
>>> after a
>>> | reboot. (I probably should have said that I've been using Debian 
>>> for my
>>> | internet server work for about 15 years: I'm not a Debian/linux 
>>> newbie,
>>> | just someone finally throwing away the reins from M$ to have a laptop
>>> | running Debian as well!)
>>>
>>> Perfect.
>>>
>>> | I get that from invoking update.packages() in an R session 
>>> launched from
>>> | a user (chris) terminal session. Before that I had done an install of
>>>
>>> I do that too, as well as calling the 'install.r' helper script from 
>>> littler
>>> (in its examples/ diretory, if interested).
>>>
>>> This requires that you make /usr/local/lib/R/site-library/ writable 
>>> for _you_
>>> as user chris, else you cannot.
>>>
>>> [ It also assumes that the Renviron setting mentioned last time is 
>>> still
>>> valid, do this:
>>>
>>> edd at max:~$ R -q -e 'print(.libPaths())'
>>> R>  print(.libPaths())
>>> [1] "/usr/local/lib/R/site-library" "/usr/lib/R/site-library"       
>>> "/usr/lib/R/library"
>>> R>
>>> R>
>>>
>>> It _must_ be first as here.  ]
>>>
>>> Now, /usr/local/lib/R/site-library/ comes shipped from the r-base-core
>>> package as owned by group 'staff' so one solution is to add yourself 
>>> to group
>>> staff:
>>>
>>> edd at max:~$ ls -ld /usr/local/lib/R/site-library/
>>> drwxrwsr-x 136 root staff 4096 2011-09-20 07:44 
>>> /usr/local/lib/R/site-library/
>>> edd at max:~$ id
>>> uid=1000(edd) gid=1000(edd)
>>> groups=1000(edd),4(adm),20(dialout),24(cdrom),27(sudo),44(video),46(plugdev),50(staff),[.....] 
>>>
>>> edd at max:~$
>>>
>>> If those two conditions are met, you should be able to run 
>>> install.packages()
>>> as you.  I still prefer it from the shell as I sometimes need 
>>> another apt-get
>>> call for a missing library etc.
>>>
>>> | I wouldn't dream of compiling locally anything I can get from a 
>>> deb you
>>> | rolled!!
>>>
>>> :-)
>>>
>>> | However, I come back to feeling that there must be three separate
>>> | problems here that come from within the packages I added after the
>>> | initial r-cran-base install. My hunch is that there are things 
>>> with XML
>>> | and rgl that I will be able to track down when restrict to just 
>>> updating
>>>
>>> "I would not dream of compiling locally" either.  You could start by 
>>> locally
>>> re-building the packages.  Which requires the build depends.
>>>
>>> | them but that at the moment there is a more global problem which I'm
>>> | guessing is caused by one package overriding the order of library 
>>> path
>>> | options seen in .libPaths() which starts with a perfectly writable 
>>> local
>>> | user directory and I am guessing there is some way that package has
>>> | insisted that the install tries /usr/lib/R/library.
>>>
>>> Agreed on the first bit.  I think /usr/lib/R/library -- where we as 
>>> users
>>> should never tread as it is /usr which belongs to apt/dpkg -- is 
>>> just the
>>> fallback.  Somehow the value got nuked.
>>>
>>> | I don't know enough about the way that R and linux work agree the 
>>> use of
>>> | those library location options. From ?.libPaths and a bit more 
>>> hunting
>>> | around the R help I had the impression that the .libPath options 
>>> would
>>> | be used in order. I think I can remove the /usr ones by hacking in
>>>
>>> Correct.
>>>
>>> | /etc/R but that presumably risks losing access to the main libraries
>>> | that will have been installed there so that doesn't look a good step
>>> | forward except perhaps to debug things.
>>>
>>> An easier option ... is to _explicitly add_ the
>>> /usr/local/lib/R/site-library/ destination as an argument to 
>>> install.packages!
>>>
>>> | I suspect that I could do a complete remove of all of R, start again
>>> | installing r-cran-base, check that R installed by that updates 
>>> fine run
>>> | by a user and then I could use synaptic to add the additional deb
>>> | packages you provide but doing that one by one and running R and
>>> | update.packages() after each one to see if I don't hit this until 
>>> afetr
>>> | a particular deb but that's going to take quite a long time so I
>>> | wondered if you or the list had quicker suggestions to debug this.
>>>
>>> That sounds too involved.
>>>
>>> | Sorry for length but hope that's clearer (got to try to be an 
>>> acceptable
>>> | friend on R lists as well as with linux and I really, really do
>>> | appreciate the generosity of this list).
>>>
>>> No sweat. We're all just building karma :)
>>>
>>> Dirk
>>>
>>>
>>
>>
>> -- 
>> Chris Evans<chris at psyctc.org>  Skype: chris-psyctc
>> Consultant Psychiatrist in Psychotherapy, Notts. PDD network;
>> Professor, Psychotherapy, Nottingham University
>> *If I am writing from one of those roles, it will be clear. Otherwise*
>> *my views are my own and not representative of those institutions    *
>> If you have difficulty Emailing me on this address or getting a reply,
>> send again but cc to:       chris dot evans at nottshc dot nhs dot uk
>> and to:                     c dot evans at nottingham dot ac dot uk
>>
>> _______________________________________________
>> R-SIG-Debian mailing list
>> R-SIG-Debian at r-project.org
>> https://stat.ethz.ch/mailman/listinfo/r-sig-debian
>>
>
>
>
>
>


-- 
Chris Evans<chris at psyctc.org>  Skype: chris-psyctc
Consultant Psychiatrist in Psychotherapy, Notts. PDD network;
Professor, Psychotherapy, Nottingham University
*If I am writing from one of those roles, it will be clear. Otherwise*
*my views are my own and not representative of those institutions    *
If you have difficulty Emailing me on this address or getting a reply,
send again but cc to:       chris dot evans at nottshc dot nhs dot uk
and to:                     c dot evans at nottingham dot ac dot uk