[R-meta] Meta-analysis of single group attitude scores

[Tommy van Steen]

Dear Michael and Wolfgang,

Thank you both very much for your helpful comments.
I have been testing the two methods suggested by Wolfgang on some fictional data to see where the differences lie, and I don’t fully understand the resulting sampling variances.

Fictional data of 2 studies:
Both are attitudes on a 7-point scale (neutral point = 4, range = 6) with n = 100.
Both studies have a mean score of 5. 
Study 1 SD = 1
Study 2 SD = 2.

With the first idea outlined by Wolfgang, the sampling variances are 0.015 (Study 1) and 0.01125 (Study 2).
With the second idea, the sampling variances are 0.000277 (Study 1) and 0.00111 (study 2).

So using the second idea, Study 2 results in a larger sampling variance, and therefore a less precise measurement, and inverse variance weighting results in a lower weight for this study. This seems to make sense as the SD for Study 2 is higher than for Study 1. 

However, using the first idea, the sampling variance of Study 2 is actually lower, which suggests that even though the SD of that study is higher, the study is more precise. 

Am I interpreting the results wrong? Or could it be that the first idea already incorporates the inverse variance calculation?

Thank you for your help!

Best wishes,
Tommy
===============
Dr Tommy van Steen
Research Associate in Psychology
University of Bath

> On 26 Nov 2017, at 10:54, Viechtbauer Wolfgang (SP) <wolfgang.viechtbauer at maastrichtuniversity.nl> wrote:
> 
> I agree that ideally one would want to use the raw mean (or raw mean minus neutral point) as the outcome measure. However, if studies differ in terms of the number of answering possibilities, then the raw values are not really comparable.
> 
> Two ideas:
> 
> 1) Divide by the SD. So you then compute d = (mean - neutral point) / SD. Then the (large-sample) sampling variance can be estimated with:
> 
> 1/n + d^2/(2*n)
> 
> 2) Divide by the possible range (not the observed one!). So you then compute d = (mean - neutral point) / range. Then the sampling variance can be estimated with:
> 
> SD^2 / (n * range^2)
> 
> Best,
> Wolfgang
> 
>> -----Original Message-----
>> From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces at r-
>> project.org] On Behalf Of Michael Dewey
>> Sent: Friday, 24 November, 2017 15:02
>> To: Tommy van Steen; r-sig-meta-analysis at r-project.org
>> Subject: Re: [R-meta] Meta-analysis of single group attitude scores
>> 
>> Dear Tommy
>> 
>> If you subtract the neutral point fr each study then this will only make
>> a difference if the studies have different neutral points but it would
>> not harm things anyway.
>> 
>> Dividing by the standard deviation seems unnecessary to me, and
>> potentially harmful. Why not use the standard error of each mean? Then
>> you would get the proper weights and an estimate of heterogeneity.
>> 
>> Michael
>> 
>> On 24/11/2017 12:26, Tommy van Steen wrote:
>>> Dear all,
>>> 
>>> I have a question about my meta-analytic approach in a project that I’m
>> working on.
>>> In this project, we are conducting a meta-analysis of attitudes. The
>> study outcome variable is the mean attitude score on a scale (often a
>> Likert-scale type survey or single item).
>>> 
>>> I am trying to figure out what the best way is to conduct a meta-
>> analysis based on this type of data as the data comes from single groups
>> and studies vary in the number of answering possibilities on the Likert-
>> scales.
>>> 
>>> My thought would be to transform the study mean into a z-score by
>> subtracting the neutral score (e.g. ‘3' in a 5-point scale study) from
>> the study mean and divide the outcome by the standard deviation of the
>> study mean. This way, the z-score reflects whether the attitude was
>> positive (f the z-score is positive) or negative (if the z-score is
>> negative), with 0 being neutral. I would then use this z-score as the
>> effect size for my meta-analysis.
>>> 
>>> Based on this, I have three questions:
>>> 1. Is this a sensible option?
>>> 2. If so, how should I calculate the study weights? (As z-scores
>> typically have an SD of 1 if I understand correctly.)
>>> 3. How would I run the meta-analysis based on these z-scores? (Simply
>> loading the scores as effect sizes in an SMD meta-analysis using the
>> metafor-package seems odd perhaps?)
>>> 
>>> Thank you very much for your time and thoughts.
>>> 
>>> With kind regards,
>>> Tommy van Steen


	[[alternative HTML version deleted]]