[R] Nested variance-covariance matrix in Multilevel model

[Tobias Guennel]

Dear R community,

I have trouble implementing a nested variance-covariance matrix in the 
lme function.

The model has two fixed effects called End and logpgc, the response 
variable is the logarithm to base 2 of  Intensity ( log2(Intensity) ) 
and the random effects are called Probe and ProbeNo.
The model has the following nesting structure: A Pixel is nested within 
the ProbeNo,the ProbeNo is within the ProbeEnd ( there are two ends for 
every probe), and the ProbeEnd is within the Probe.

Now the problem I have is that the variance-covariance structure of the 
model is quite complex and I can not find the right syntax for fitting 
it in the lme function.

The variance-covariance structure  is a block diagonal matrix of the form,
          V1   0       0
 V=    0      V2    0
          0      0        V3

where V1...V3 are of the structure:
          v11      v12
  V1=                          and so on.
          v21      v22

V1...V3 are assumed to have a compound symmetric variance-covariance 
structure and therefore the submatrices are of the form:
                   Lambda   Delta1       Delta1 ...   Delta1
                   Delta1      Lambda    Delta1 ...   Delta1
v11=v22=    .......

                   Delta1   .....                               Lambda
                  
                   Delta2      Delta2       Delta2 ...   Delta2
                   Delta2      Delta2       Delta2 ...   Delta2
v12=v21=    .......

                   Delta2   .....                               Delta2

The elements of these submatrices depend only upon the three covariance 
parameters: the compound symmetry parameter delta; the variance of 
random effect sigma^2g; and the residual variance sigma^2. I have 
formulas for the submatrices Lambda,Delta1 and Delta2 which I can't 
really paste in here.

The SAS code dealing with this model is the following:

proc mixed data=rnadeg.pnau;
title 'CV structure for PNAU';
class probepos probeno end probe pixelid newprobeid;
model logPM=end logpgc / ddfm=satterth;
random probeno newprobeid / subject=probe type=cs;
lsmeans end / diff cl; run;

Any ideas are appreciated a lot since I am kind of stuck at this point.

Thank you
Tobias Guennel