[R] individual scales in random subset of pairwise distance survey

[context grey]

Hello,

I'm curious if anyone has encounted a version of this
problem
(and it's solution) involving finding a consistent set
of scales
for subsets of survey data.

The goal is to obtain peoples' rankings of pairwise
similarity of a large
number of items, on a 1..5 scale for example, and
average these
across people to use as input to MDS:
  How similar is object A to B    on a 1..5 scale ___
  How similar is object A to C    on a 1..5 scale ___
etc.
  
Because there are many items, there are N(N-1)/2
pairs, so it is not
practical to show every pair to everyone.   Showing
people the
pairs corresponding to random subsets of the objects
seems desirable.

THe problem is that, a particular random subset might
by chance
contain objects that would all be rated "5" if one
were to see
the entire dataset.  When ranking pairs from this
subset, the scale
of 1..5 is different.  

If we ensure that each pair of people must see some
data in common,
then one can think about obtaining a set of scales,
one for each
person, that causes the data that is commonly ranked
to have
as similar scores as possible, summed across all pairs
of people.


Please let me know if you know of a standard procedure
for
this or any similar problems.    

Thank you.