[R] procrustes results affected by order of sites in input file

[Gavin Simpson]

On Thu, 2010-12-02 at 11:19 -0600, Christine Dolph wrote:
> Hi, Thanks very much for your response.

Thanks Christy,

Apologies if I sounded off-hand or dismissive yesterday. It was a busy
day, and as your mail lacked a reproducible example nor the code you
ran, I wanted to deal with the low-hanging fruit before spending time
looking at the issue.

I am currently running a simulation on one of the in-built datasets to
see if I can replicate your findings.

Jari has mentioned, privately, that isoMDS() can be order dependent if
there are ties in the dissimilarities. What, if anything, this has to do
with the protest issue you report is not clear, hence the simulation I
am running.

I will report back later when the simulation has finished... which could
be some time if I continued to code bone-headedly this morning!

Finally, thanks for raising this issue with us.

All the best,

G

> Unfortunately, using the set.seed() call does not seem to solve my problem.
> If I do not use set.seed(), I do indeed get some small differences in
> protest() results due to the effect of random starts. But with my sites in a
> given order in the input files, and using set.seed(), I get the same results
> every time I run protest().
> 
> It's only when I change the order of sites in the input files that I get a
> big change in results. The differences I see in protest() results after
> re-sorting the order of sites in the input file is quite large, larger than
> the smaller differences do to random starts I see if I run protest() without
> the set.seed. When sites are re-ordered in the input file, the
> correlation-like statistic produced by protest() can jump from between 0.28
> to 0.52 for a dataset with 38 sites, with a corresponding change in
> significance. Without using set.seed the numbers do jump around a bit, but
> not to the same extreme degree.
> 
> Looking at the NMDS plots to see how ordinations change with the order of
> sites in the file, it seems that if I re-order sites, I am simply getting
> ordination solutions that are reflections on one another across one of the 3
> NMDS axes. These are not real differences in the ordination solutions. Is
> there something I am failing to include in the protest() call that should
> enable the function to flip one ordination around to maximize it's
> similarity to the other ordination (i.e., that should orient the two
> ordinations the same way in 3-dimensional space)?
> 
> Thanks,
> 
> --Christy
> 
> 
> On Thu, Dec 2, 2010 at 6:02 AM, Gavin Simpson <gavin.simpson at ucl.ac.uk>wrote:
> 
> > On Wed, 2010-12-01 at 14:19 -0600, Christine Dolph wrote:
> > > Dear All,
> > >
> > > I am using a Procrustes analysis to compare two NMDS ordinations for the
> > > same set of sites. One ordination is based on fish data, the other is
> > based
> > > on invertebrate data. Ordinations were derived using metaMDS() from the
> > > {vegan} library as follows:
> > >
> > > fish.mds<-metaMDS(fish.data, distance="bray", k=3, trymax=100,
> > > wascores=TRUE, trace=TRUE, zero="add")
> > >
> > > invert.mds<-metaMDS(bugcal.a, distance="bray", k=3, trymax=100,
> > > wascores=TRUE, trace=TRUE, zero="add"),
> > >
> > >
> > > where fish.data and invert.data are site-by-abundance matrices for fish
> > and
> > > invertebrates, respectively.
> > >
> > > I have then used protest() to test the significance between the two
> > > ordinations:
> >
> > Simple things first:
> >
> > You did have a set.seed() call before the protest() call didn't you? In
> > fact, I'd probably put set.seed() calls before the two metaMDS() calls
> > as well.
> >
> > The ordering should have no impact at all, but the random starts in
> > metaMDS() and the random permutations in protest() will vary from run to
> > run unless you set the random seed to some known value.
> >
> > G
> >
> > > protest.results<-protest(fish.mds, invert.mds, scale=TRUE,
> > symmetric=TRUE,
> > > permutations=999)
> > >
> > > The problem is, the results of the protest analysis seem to be affected
> > by
> > > the original ordering of sites in the data input files. That is, if I
> > > re-sort one of the input files based on some criteria, I sometimes see a
> > > change in the strength and significance of the concordance results. In an
> > > attempt to figure out what is happening, I have re-ordered each dataset
> > in a
> > > number of ways, and plotted the resulting ordinations. I have seen that
> > > while the ordering of sites in the input file does not change the spatial
> > > relationship between them (i.e., does not change the fundamental
> > ordination
> > > solution), it can change how the sites are oriented with respect to the
> > > different NMDS axes. I was of the belief that a difference in orientation
> > > with respect to the NMDS axes should not affect the Procrustes comparison
> > of
> > > two ordinations, as the protest function should be rotating and
> > reflecting
> > > one matrix with respect to the other to find the closest match between
> > them
> > > (i.e., it should be accounting for differences in how the two solutions
> > are
> > > oriented in space). However, I appear to see different results depending
> > on
> > > how sites are oriented with respect to each NMDS axis.
> > >
> > > When comparing two ordination solutions with Protest, is it necessary to
> > > ensure that the original data input files are ordered in the same way?
> > >
> > > Thanks in advance for any insight.
> > >
> > > Sincerely,
> > >
> > >
> > >
> >
> > --
> > %~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%
> >  Dr. Gavin Simpson             [t] +44 (0)20 7679 0522
> >  ECRC, UCL Geography,          [f] +44 (0)20 7679 0565
> >  Pearson Building,             [e] gavin.simpsonATNOSPAMucl.ac.uk
> >  Gower Street, London          [w] http://www.ucl.ac.uk/~ucfagls/<http://www.ucl.ac.uk/%7Eucfagls/>
> >  UK. WC1E 6BT.                 [w] http://www.freshwaters.org.uk
> > %~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%
> >
> >
> 
> 

-- 
%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%
 Dr. Gavin Simpson             [t] +44 (0)20 7679 0522
 ECRC, UCL Geography,          [f] +44 (0)20 7679 0565
 Pearson Building,             [e] gavin.simpsonATNOSPAMucl.ac.uk
 Gower Street, London          [w] http://www.ucl.ac.uk/~ucfagls/
 UK. WC1E 6BT.                 [w] http://www.freshwaters.org.uk
%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%