[R] have to point it out again: a distribution question
bogdan romocea
br44114 at gmail.com
Fri Apr 29 20:30:04 CEST 2005
> Then, Reid, or other r-gurus, is there a good way to descritize
> the sample into 3 category: 2 tails and the body?
Out of curiosity, how do you plan to use that information? What would
you do if you knew that the 'body' starts here and ends there?
-----Original Message-----
From: WeiWei Shi [mailto:helprhelp at gmail.com]
Sent: Thursday, April 28, 2005 4:18 PM
To: Huntsinger, Reid
Cc: R-help at stat.math.ethz.ch
Subject: Re: [R] have to point it out again: a distribution question
Here is summary of
l<-qqnorm(kk) # kk is my sample
l$y (which is my sample)
l$x (which is therotical quantile)
diff<-l$y-l$x
and
> summary(l$y)
Min. 1st Qu. Median Mean 3rd Qu. Max.
0.9007 0.9942 0.9998 0.9999 1.0060 1.1070
> summary(l$x)
Min. 1st Qu. Median Mean 3rd Qu. Max.
-4.145e+00 -6.745e-01 0.000e+00 2.383e-17 6.745e-01 4.145e+00
> summary(diff)
Min. 1st Qu. Median Mean 3rd Qu. Max.
-3.0380 0.3311 0.9998 0.9999 1.6690 5.0460
Comparing diff with l$x, though the 1st Qu. and 3rd Qu. are different,
diff and l$x seem similar to each other, which are proved by
qqnorm(l$x) and qqnorm(diff).
running the following codes:
r<-rnorm(1000)+1 # since my sample shift from zero to 1
qq(r[r>0.9 & r<1.2]) # select the central part
this gives me a straight line now.
Thanks for the good explanation for the phenomena.
Then, Reid, or other r-gurus, is there a good way to descritize the
sample into 3 category: 2 tails and the body?
Thanks again,
Weiwei
On 4/28/05, Huntsinger, Reid <reid_huntsinger at merck.com> wrote:
> Stock returns and other financial data have often found to be heavy-tailed.
> Even Cauchy distributions (without even a first absolute moment) have been
> entertained as models.
>
> Your qq function subtracts numbers on the scale of a normal (0,1)
> distribution from the input data. When the input data are scaled so that
> they are insignificant compared to 1, say, then you get essentially the
> "theoretical quantiles" ie the "x" component of the list back from l$x -
> l$y. l$x is basically a sample from a normal(0,1) distribution so they do
> line up perfectly in the second qqnorm(). Is that what's happening?
>
> Reid Huntsinger
>
>
> -----Original Message-----
> From: r-help-bounces at stat.math.ethz.ch
> [mailto:r-help-bounces at stat.math.ethz.ch] On Behalf Of WeiWei Shi
> Sent: Thursday, April 28, 2005 1:38 PM
> To: Vincent ZOONEKYND
> Cc: R-help at stat.math.ethz.ch
> Subject: [R] have to point it out again: a distribution question
>
> Dear R-helpers:
> I pointed out my question last time but it is only partially solved.
> So I would like to point it out again since I think it is very
> interesting, at least to me.
> It is a question not about how to use R, instead it is a kind of
> therotical plus practical question, represented by R.
>
> I came with this question when I built model for some stock returns.
> That's the reason I cannot post the complete data here. But I would
> like to attach some plots here (I zipped them since the original ones
> are too big).
>
> The first plot qq1, is qqnorm plot of my sample, giving me some
> "S"-shape. Since I am not very experienced, I am not sure what kind of
> distribution my sample follows.
>
> The second plot, qq2, is obtained via
> qqnorm(rt(10000, 4)) since I run
> fitdistr(kk, 't') and got
> m s df
> 9.998789e-01 7.663799e-03 3.759726e+00
> (5.332631e-05) (5.411400e-05) (8.684956e-02)
>
> The second plot seems to say my sample distr follows t-distr. (not sure of
> this)
>
> BTW, what the commands for simulating other distr like log-norm,
> exponential, and so on?
>
> The third one was obtained by running the following R code:
>
> Suppose my data is read into dataset k from file "f392.txt":
> k<-read.table("f392.txt", header=F) # read into k
> kk<-k[[1]]
> qq(kk)
>
> qq function is defined as below:
> qq<-function(dataset){
> l<-qqnorm(dataset, plot.it=F)
> diff<-l$y-l$x # difference b/w sample and it's therotical quantile
> qqnorm(diff)
> }
>
> The most interesting thing is (if there is not any stupid game here,
> and if my sample follows some kind of distribution (no matter if such
> distr has been found or not)), my qq function seems like a way to
> evaluate it. But what I am worried about, the line is too "perfect",
> which indiates there is something goofy here, which can be proved via
> some mathematical inference to get it. However I used
> qq(rnorm(10000))
> qq(rt(10000, 3.7)
> qq(rf(....))
>
> None of them gave me this perfect line!
>
> Sorry for the long question but I want to make it clear to everybody
> about my question. I tried my best :)
>
> Thanks for your reading,
>
> Weiwei (Ed) Shi, Ph.D
>
> On 4/23/05, Vincent ZOONEKYND <zoonek at gmail.com> wrote:
> > If I understand your problem, you are computing the difference between
> > your data and the quantiles of a standard gaussian variable -- in
> > other words, the difference between the data and the red line, in the
> > following picture.
> >
> > N <- 100 # Sample size
> > m <- 1 # Mean
> > s <- 2 # dispersion
> > x <- m + s * rt(N, df=2) # Non-gaussian data
> >
> > qqnorm(x)
> > abline(0,1, col="red")
> >
> > And you get
> >
> > y <- sort(x) - qnorm(ppoints(N))
> > hist(y)
> >
> > This is probably not the right line (not only because your mean is 1,
> > the slope is wrong as well -- if the data were gaussian, you could
> > estimate it with the standard deviation).
> >
> > You can use the "qqline" function to get the line passing throught the
> > first and third quartiles, which is probably closer to what you have
> > in mind.
> >
> > qqnorm(x)
> > abline(0,1, col="red")
> > qqline(x, col="blue")
> >
> > The differences are
> >
> > x1 <- quantile(x, .25)
> > x2 <- quantile(x, .75)
> > b <- (x2-x1) / (qnorm(.75)-qnorm(.25))
> > a <- x1 - b * qnorm(.25)
> > y <- sort(x) - (a + b * qnorm(ppoints(N)))
> > hist(y)
> >
> > And you want to know when the differences ceases to be "significantly"
> > different from zero.
> >
> > plot(y)
> > abline(h=0, lty=3)
> >
> > You can use the plot fo fix a threshold, but unless you have a model
> > describing how non-gaussian you data are, this will be empirical.
> >
> > You will note that, in those simulations, the differences (either
> > yours or those from the lines through the first and third quartiles)
> > are not gaussian at all.
> >
> > -- Vincent
> >
> >
> > On 4/22/05, WeiWei Shi <helprhelp at gmail.com> wrote:
> > > hope it is not b/c some central limit therory, otherwise my initial
> > > plan will fail :)
> > >
> > > On 4/22/05, WeiWei Shi <helprhelp at gmail.com> wrote:
> > > > Hi, r-gurus:
> > > >
> > > > I happened to have a question in my work:
> > > >
> > > > I have a dataset, which has only one dimention, like
> > > > 0.99037297527605
> > > > 0.991179836732708
> > > > 0.995635340631367
> > > > 0.997186769599305
> > > > 0.991632565640424
> > > > 0.984047197106486
> > > > 0.99225943762649
> > > > 1.00555642128421
> > > > 0.993725402926564
> > > > ....
> > > >
> > > > the data is saved in a file called f392.txt.
> > > >
> > > > I used the following codes to play around :)
> > > >
> > > > k<-read.table("f392.txt", header=F) # read into k
> > > > kk<-k[[1]]
> > > > l<-qqnorm(kk)
> > > > diff=c()
> > > > lenk<-length(kk)
> > > > i=1
> > > > while (i<=lenk){
> > > > diff[i]=l$y[i]-l$x[i] # save the difference of therotical quantile
> > > > and sample quantile
> > > > # remember, my sample mean is around 1
> > > > while the therotical one, 0
> > > > i<-i+1
> > > > }
> > > > hist(diff, breaks=300) # analyze the distr of such diff
> > > > qqnorm(diff)
> > > >
> > > > my question is:
> > > > from l<-qqnorm(kk), I wanted to know, from which point (or cut), the
> > > > sample points start to become away from therotical ones. That's the
> > > > reason I played around the "diff" list, which gives me the difference.
> > > > To my surprise, the diff is perfectly normal. I tried to use some
> > > > kk<-c(1, 2, 5, -1 , ...) to test, I concluded it must be some
> > > > distribution my sample follows gives this finding.
> > > >
> > > > So, any suggestion on the distribution of my sample? I think there
> > > > might be some mathematical inference which can leads this observation,
> > > > but not quite sure.
> > > >
> > > > btw,
> > > > > fitdistr(kk, 't')
> > > > m s df
> > > > 9.999965e-01 7.630770e-03 3.742244e+00
> > > > (5.317674e-05) (5.373884e-05) (8.584725e-02)
> > > >
> > > > btw2, can anyone suggest a way to find the "cut" or "threshold" from
> > > > my sample to discretize them into 3 groups: two tail-group and one
> > > > main group.--------- my focus.
> > > >
> > > > Thanks,
> > > >
> > > > Ed
> > > >
> > >
> > > ______________________________________________
> > > R-help at stat.math.ethz.ch mailing list
> > > https://stat.ethz.ch/mailman/listinfo/r-help
> > > PLEASE do read the posting guide!
> > > http://www.R-project.org/posting-guide.html
> > >
> >
>
> ------------------------------------------------------------------------------
> Notice: This e-mail message, together with any attachment...{{dropped}}
______________________________________________
R-help at stat.math.ethz.ch mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
More information about the R-help
mailing list