[R] NFFT on a Zoo?

[Mike Marchywka]

----------------------------------------
> Date: Fri, 5 Nov 2010 00:14:15 -0700
> From: FlyMyPG at gmail.com
> To: marchywka at hotmail.com
> CC: ggrothendieck at gmail.com; r-help at r-project.org; rpy-list at lists.sourceforge.net
> Subject: Re: [R] NFFT on a Zoo?
>
> FWIW: It turns out I dove into a rabbit hole:
>
> 1. Though the gaps in my 3-axis accelerometer data represent 10% data
> loss (OMG!), the number of gaps represents only 0.1% of the 3 million
> data points (BFD).
>
> 2. The data is noisy enough that 0.1% discontinuity can't affect an
> FFT. Each gap was removed simply by adjusting subsequent timestamps.
>
> 3. With the gaps removed, the remaining jitter in the timestamps is both
> small and nearly normally distributed (no systematic errors). So the
> timestamps were eliminated from further processing, and the mean
> inter-sample time was used as the sampling period.
>
> So, neither NFFT nor Zoo are needed, since a regular FFT now works just
> fine.

Well, again, it is easy to simulate these effects in R by sampling
a sine wave or other known signal at the wrong times, DFT, and see
what spectrum looks like. I suggested sine wave to start, but any
nonlinearities can be hard to estimate without a little work. You
can also do two-tone tests by hand and see IMD, harmonics, etc.


>
> The Moral of the Story is: "Take a closer look at the data before
> deciding difficult processing is needed."
>
> Homer Simpson translation: "Doh!"
>
I think Bart said " if it is in a book it must be true" and computers
of course don't make mistakes :)


> A big "Thanks!" to all who responded to my newbie posts: The R
> Community is richly blessed with wisdom, kindness and patience.

This is the kind of thing you can play with on the back of an 
envelope when bored once you get started. 

>
> -BobC
>
>
>
> On 11/03/2010 01:22 PM, Mike Marchywka wrote:
> > ----------------------------------------
> >
> >> From: ggrothendieck at gmail.com
> >> Date: Wed, 3 Nov 2010 15:27:13 -0400
> >> To: FlyMyPG at gmail.com
> >> CC: r-help at r-project.org; rpy-list at lists.sourceforge.net
> >> Subject: Re: [R] NFFT on a Zoo?
> >>
> >> On Wed, Nov 3, 2010 at 2:59 PM, Bob Cunningham wrote:
> >>
> >>> I have an irregular time series in a Zoo object, and I've been unable to
> >>> find any way to do an FFT on it. More precisely, I'd like to do an NFFT
> >>> (non-equispaced / non-uniform time FFT) on the data.
> >>>
> >>> The data is timestamped samples from a cheap self-logging accelerometer.
> >>> The data is weakly regular, with the following characteristics:
> >>> - short gaps every ~20ms
> >>> - large gaps every ~200ms
> >>> - jitter/noise in the timestamp
> >>>
> >>> The gaps cover ~10% of the acquisition time. And they occur often enough
> >>> that the uninterrupted portions of the data are too short to yield useful
> >>> individual FFT results, even without timestamp noise.
> >>>
> >>> My searches have revealed no NFFT support in R, but I'm hoping it may be
> >>> known under some other name (just as non-uniform time series are known as
> >>> 'zoo' rather than 'nts' or 'nuts').
> >>>
> >>> I'm using R through RPy, so any solution that makes use of numpy/scipy would
> >>> also work. And I care more about accuracy than speed, so a non-library
> >>> solution in R or Python would also work.
> >>>
> >>> Alternatively, is there a technique by which multiple FFTs over smaller
> >>> (incomplete) data regions may be combined to yield an improved view of the
> >>> whole? My experiments have so far yielded only useless results, but I'm
> >>> getting ready to try PCA across the set of partial FFTs.
> >>>
> >>>
> >>
> >
> > I'm pretty sure all of this is in Oppenheim and Shaffer meaning it
> > is also in any newer books. I recall something about averaging
> > but you'd need to look at details. Alternatively, and this is from
> > distant memory so maybe someone else can comment, you can just
> > feed a regularly spaced time series to anyone, go get FFTW for example,
> > and insert zeroes for missing data. This is equivalent to multiplying
> > your real data with a window function that is zero at missing points.
> > I think you can prove that multiplication
> > in time domain is convolution in FT domain so you can back this out
> > by deconvolving with your window function spectrum. This probably is not
> > painless, the window spectrum will have badly placed zeroes etc, but it
> > may be helpful.
> > Apaprently this is still a bit of an open issue,
> >
> > http://books.google.com/books?id=BW1PdOqZo6AC&pg=PA2&lpg=PA2&dq=dft+window+missing+data&source=bl&ots=fSY-iRoCNN&sig=30cC0SdkrDcp62iWc-Mv26mfNjI&hl=en&ei=AMTRTNmyMYP88AauxtzKDA&sa=X&oi=book_result&ct=result&resnum=6&ved=0CDEQ6AEwBTgK#v=onepage&q&f=false
> >
> >
> >
> > You should be able to do the case of a sine wave with pencil and paper
> > and see if or how this really would work.
> >
> >
> >
> >> Check out the entire thread that starts here.
> >>
> >> http://www.mail-archive.com/r-help@r-project.org/msg36349.html
> >>
> >> --
> >> Statistics& Software Consulting
> >> GKX Group, GKX Associates Inc.
> >> tel: 1-877-GKX-GROUP
> >> email: ggrothendieck at gmail.com
> >>





- - - - - -

Mike Marchywka | V.P. Technology

415-264-8477
marchywka at phluant.com

Online Advertising and Analytics for Mobile
http://www.phluant.com