[R] S4 / operator "[" : Compatibility issue between lme4 and kml

[Martin Maechler]

>>>>> Christophe Genolini <cgenolin at u-paris10.fr>
>>>>>     on Fri, 5 Jun 2015 00:36:42 -0700 writes:

    > Hi all,
    > There is a compatibility issue between the package 'lme4' and my package
    > 'kml'. I define the "[" operator. It works just fine in my package (1). If I
    > try to use the lme4 package, then it does no longer work (2). Moreover, it
    > has some kind of strange behavior (3). Do you know what is wrong? Any idea
    > of how I can correct that?

    > Here is a reproductible example, and the same code with the result follows.

    > Thanks for your help
    > Christophe

  [ ... I'm providing slightly different code below .... ]

> --- 8< ----------------- Execution of the previous code -------------------

> > library(kml)
> Le chargement a nécessité le package : clv
> Le chargement a nécessité le package : cluster
> Le chargement a nécessité le package : class
> Le chargement a nécessité le package : longitudinalData
> Le chargement a nécessité le package : rgl
> Le chargement a nécessité le package : misc3d
> > dn <- gald(1)

>  ###########
> ### (1) the "[" operator works just fine

> > dn["traj"]
>       t0   t1    t2    t3    t4   t5   t6    t7    t8    t9   t10
> i1 -3.11 4.32  2.17  1.82  4.90 7.34 0.83 -2.70  5.36  4.96  3.16
> i2 -7.11 1.40 -2.40 -2.96  4.31 0.50 1.25  0.52 -0.04  7.55  5.50
> i3  2.80 6.23  6.08  2.87  2.58 2.88 6.58 -2.38  2.30 -1.74 -3.23
> i4  2.24 0.91  6.50 10.92 11.32 7.79 7.78 10.69  9.15  1.07 -0.51

>  ###########
> ### (2) using 'lme4', it does no longer work

> > library(lme4)
> Le chargement a nécessité le package : Matrix
> Le chargement a nécessité le package : Rcpp
> > dn["traj"]
> Error in x[i, j] :
>   erreur d'évaluation de l'argument 'j' lors de la sélection d'une méthode
> pour la fonction '[' : Erreur : l'argument "j" est manquant, avec aucune
> valeur par défaut

>  ###########
> ### (3) If I define again the "[", it does not work the first time I call
> it, but it work the second time!
> > setMethod("[",
> +   signature=signature(x="ClusterLongData", i="character", j="ANY",drop="ANY"),
> +   definition=function (x, i, j="missing", ..., drop = TRUE){
> +       x <- as(x, "LongData")
> +       return(x[i, j])
> +     }
> + )
> [1] "["

> ### No working the first time I use it
> > dn["traj"]
> Error in dn["traj"] :
>   l'argument "j" est manquant, avec aucune valeur par défaut

> ### But working the second time
> > dn["traj"]
>       t0   t1    t2    t3    t4   t5   t6    t7    t8    t9   t10
> i1 -3.11 4.32  2.17  1.82  4.90 7.34 0.83 -2.70  5.36  4.96  3.16
> i2 -7.11 1.40 -2.40 -2.96  4.31 0.50 1.25  0.52 -0.04  7.55  5.50
> i3  2.80 6.23  6.08  2.87  2.58 2.88 6.58 -2.38  2.30 -1.74 -3.23
> i4  2.24 0.91  6.50 10.92 11.32 7.79 7.78 10.69  9.15  1.07 -0.51 

I have made some investigations, but have to stop for now, and
leave this hopefully to others knowledgable about S4 method
dispatch, etc :

1) I am confident to say that you have uncovered an "unfelicity if
  not a bug" in R.

2) I am also pretty confident that the "[" methods that you
  define in 'kml' and in the package '

3) Diagnosing is not easy: As you have shown yourself above,
  in some situations the bug "bites" and if you repeat the *same*
  code, things work.

  This is related to the fact that S4 methods are __cached__
  (so next time they are found more quickly) under some
  circumstances, and the cache is cleared under other such circumstances.

3b) Actually, I am sure that we have seen +/- the same problem many
    months ago, in other contexts but did not get "down to it";
    and at the moment, I cannot quickly find where to look for
    the problem there...


##--- 8< ------------Commented (incl output) reproducible code--------------
library(kml)

### Creating some data
dn <- gald(1)
(dnt <- dn["traj"])

showMethods("[")
## Function: [ (package base)
## x="ClusterLongData", i="character"
## x="ListPartition", i="ANY"
## x="LongData", i="ANY"
## x="LongData", i="character"
##     (inherited from: x="LongData", i="ANY")
## x="LongData3d", i="ANY"
## x="nonStructure", i="ANY"
## x="ParChoice", i="ANY"
## x="ParKml", i="ANY"
## x="ParLongData", i="ANY"
## x="Partition", i="ANY"
## x="ParWindows", i="ANY"

### using Matrix  (or lme4, which 'Depends' on Matrix; hence same effect)
library(Matrix)
dn["traj"]
## Error in x[i, j] :
##   error in evaluating the argument 'j' in selecting a method for function '[': Error: argument "j" is missing, with no default
traceback()
## 3: x[i, j]
## 2: dn["traj"]
## 1: dn["traj"]
(ms <- methods(`[`)) ## 81 methods

##---- MM: debugging :
trace("[", browser, signature=c("ClusterLongData", "character", "missing",   "missing"))
trace("[", browser, signature=c("LongData",        "character", "character", "missing"))
dn["traj"]
## -> you get into the browser, just press   "c"   twice (once for each "trace")
## ==> it works !!

## Remove the tracing :
untrace("[", signature=c("ClusterLongData", "character", "missing",   "missing"))
untrace("[", signature=c("LongData",        "character", "character", "missing"))
dn["traj"]
## Error in dn["traj"] : argument "j" is missing, with no default

## Debugging only the *inner* function:
trace("[", browser, signature=c("LongData",        "character", "character", "missing"))
dn["traj"]
## Error ....
untrace("[", signature=c("LongData",        "character", "character", "missing"))

## Debugging only the *outer* function:
trace("[", browser, signature=c("ClusterLongData", "character", "missing",   "missing"))
dn["traj"] ## -> debugger, press 'c'
## it works!
##       t0    t1    t2   t3   t4   t5   t6    t7    t8    t9   t10
## i1  7.38  4.80  4.80 0.73 0.58 2.22 0.55 -1.05  0.79  5.20  4.43
## i2  1.55  2.01 -0.29 2.12 4.44 7.33 9.09  4.76 12.18  5.92  8.06
## i3 13.60 14.72 10.15 9.25 8.70 8.34 6.71  5.84  3.44  2.10  2.17
## i4 -7.49 -1.80  0.08 2.51 6.61 4.56 8.96  3.05  3.41 -2.62 -4.09
untrace("[", signature=c("ClusterLongData", "character", "missing",   "missing"))

##--- {end of reproducible code} --------------