[R-meta] confidence interval very large

[James Pustejovsky]

That's a good question. No, the same concerns apply to CIs and to SEs and
t-statistics.

For F-statistics our research (Tipton & Pustejovsky, 2015 in JEBS) found
that F-tests tend to be conservative, with lower-than-nominal Type-I error,
when df are very low. So they're not necessarily problematic in terms of
Type I error control, but they do tend to have very low power so still
wouldn't put much stock in them.

On Wed, Jul 27, 2022 at 3:41 PM Huang <wuhuang0421 using gmail.com> wrote:

> Thanks so much for your responses. Dr.Pustejovsky, besides confidence
> interval, are other results (e.g., robust SE and t-tests & F-tests
> trustable) when df < 4? I appreciate your help.
>
> Best wishes,
> Huang
>
> On Wed, Jul 27, 2022 at 11:51 AM James Pustejovsky <jepusto using gmail.com>
> wrote:
>
>> Hi Huang,
>>
>> The wide confidence interval is due to having only one degree of freedom
>> (a t distribution with 1 degree of freedom is equivalent to the dreaded,
>> ill-behaved Cauchy distribution). I would not trust confidence intervals
>> based on so few degrees of freedom. Tipton (2015 Psych Methods) suggests
>> disregarding results based on RVE if df < 4.
>>
>> Generally, low degrees of freedom means that there is very little
>> information available to estimate standard errors--i.e., the SEs will be
>> very noisy. This can happen, for instance, if you are comparing two
>> different categories of effect sizes and one of the categories only appears
>> in two studies.
>>
>> You could, in these instances, report the model-based confidence
>> interval, but in doing so you're relying on all of the additional
>> assumptions of the model (such as homoskedasticity of random effects,
>> correctness of correlations between sampling errors). So you would need to
>> be clear about these caveats. You might consider also reporting sensitivity
>> analyses under some range of plausible alternative assumptions.
>>
>> James
>>
>> On Tue, Jul 26, 2022 at 10:39 PM Huang <wuhuang0421 using gmail.com> wrote:
>>
>>> Hi folks,
>>>
>>> I am using conf_int to calculate the confidence interval but the results
>>> showed the confidence interval are very large. For example, I have a mean
>>> estimate 0.1655 with a SE of 0.3014 (df = 1), but the 95% confidence
>>> interval is [-3.659, 3.990]. Is this correct? Should I report the model
>>> based confidence interval instead of RVE based? Thank you so much for
>>> taking time to respond to my email.
>>>
>>> Best wishes,
>>> HUang
>>>
>>>
>>> >
>>> >
>>>
>>>         [[alternative HTML version deleted]]
>>>
>>> _______________________________________________
>>> R-sig-meta-analysis mailing list @ R-sig-meta-analysis using r-project.org
>>> To manage your subscription to this mailing list, go to:
>>> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>>>
>>

	[[alternative HTML version deleted]]