[R] Hierarchical data sets: which software to use?

[Anton du Toit]

Hi Douglas,

Thanks for your helpful response. I've commented on some of the points
you raised below:

> Although this may not be helpful for your immediate goal, storing and
manipulating data of this size and complexity (and, I expect, cost for
collection) really calls for tools like relational databases.  A
single flat file of 2500 variables by 1500 cases is almost never the
best way to organize such data.  A normalized representation as a
collection of interlinked tables in a relational data base is much
more effective and less error prone.  The widespread use of
spreadsheets or SPSS data sets or SAS data sets which encourage the
"single table with a gargantuan number of columns, most of which are
missing data in most cases" approach to organization of longitudinal
data is regrettable.

I'm both relieved and daunted by this. Daunted because it means I'll
need to learn another package (probably postGreSQL or MySQL?), but
relieved because constructing a 2500 by 1500 file seemed intuitively
wrong, as well as introducing the possibility of errors
unnecessarily--surely it makes more sense to leave the data as is.

As far as immediate goals go--I am at the beginning of a thesis, and I
have more research planned after that, so I want to get things right
from the  start.


> For later analysis in R it is better to start with "long" form of the
data, as opposed to the "wide" form, even if it means repeating
demographic information over several occasions.  Using a relational
database allows for a long view to be generated without the
possibility of inconsistency in the demographics.  I am using the
descriptions "long" and "wide" in the sense that they are used in the
reshape help page.  See

?reshape

> in R.  The long view is also called the subject/occasion view in the
sense that each row corresponds to one subject on one occasion.

> Robert Gentleman's book "R Programming for Bioinformatics" provides
background on linking R to relational databases.

Thanks--I'll look this one up.


> As I said at the beginning, you may not want to undertake the
necessary study and effort to reorganize your data for this specific
project but if you do this a lot you may want to consider it.

As above: a stitch in time, I suppose.

Thanks again.

Anton

On Sat, Feb 6, 2010 at 3:22 AM, Douglas Bates <bates at stat.wisc.edu> wrote:
> On Sun, Jan 31, 2010 at 10:24 PM, Anton du Toit <atdutoitrhelp at gmail.com> wrote:
>> Dear R-helpers,
>>
>> I’m writing for advice on whether I should use R or a different package or
>> language. I’ve looked through the R-help archives, some manuals, and some
>> other sites as well, and I haven’t done too well finding relevant info,
>> hence my question here.
>>
>> I’m working with hierarchical data (in SPSS lingo). That is, for each case
>> (person) I read in three types of (medical) record:
>>
>> 1. demographic data: name, age, sex, address, etc
>>
>> 2. ‘admissions’ data: this generally repeats, so I will have 20 or so
>> variables relating to their first hospital admission, then the same 20 again
>> for their second admission, and so on
>>
>> 3. ‘collections’ data, about 100 variables containing the results of a
>> battery of standard tests. These are administered at intervals and so this
>> is repeating data as well.
>>
>> The number of repetitions varies between cases, so in its one case per line
>> format the data is non-rectangular.
>>
>> At present I have shoehorned all of this into SPSS, with each case on one
>> line. My test database has 2,500 variables and 1,500 cases (or persons), and
>> in SPSS’s *.SAV format is ~4MB. The one I finally work with will be larger
>> again, though likely within one order of magnitude. Down the track, funding
>> permitting, I hope to be working with tens of thousands of cases.
>
> Although this may not be helpful for your immediate goal, storing and
> manipulating data of this size and complexity (and, I expect, cost for
> collection) really calls for tools like relational databases.  A
> single flat file of 2500 variables by 1500 cases is almost never the
> best way to organize such data.  A normalized representation as a
> collection of interlinked tables in a relational data base is much
> more effective and less error prone.  The widespread use of
> spreadsheets or SPSS data sets or SAS data sets which encourage the
> "single table with a gargantuan number of columns, most of which are
> missing data in most cases" approach to organization of longitudinal
> data is regrettable.
>
> For later analysis in R it is better to start with "long" form of the
> data, as opposed to the "wide" form, even if it means repeating
> demographic information over several occasions.  Using a relational
> database allows for a long view to be generated without the
> possibility of inconsistency in the demographics.  I am using the
> descriptions "long" and "wide" in the sense that they are used in the
> reshape help page.  See
>
> ?reshape
>
> in R.  The long view is also called the subject/occasion view in the
> sense that each row corresponds to one subject on one occasion.
>
> Robert Gentleman's book "R Programming for Bioinformatics" provides
> background on linking R to relational databases.
>
>
> As I said at the beginning, you may not want to undertake the
> necessary study and effort to reorganize your data for this specific
> project but if you do this a lot you may want to consider it.
>
>> I am wondering if I should keep using SPSS, or try something else.
>>
>> The types of analysis I’ll typically will have to do will involve comparing
>> measurements at different times, e.g. before/ after treatment. I’ll also
>> need to compare groups of people, e.g. treatment / no treatment. Regression
>> and factor analyses will doubtless come into it at some point too.
>>
>> So:
>>
>> 1. should I use R or try something else?
>>
>> 2. can anyone advise me on using R with the type of data I’ve described?
>>
>>
>> Many thanks,
>>
>> Anton du Toit
 ______________________________________________
>> R-help at r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-help
>> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
>> and provide commented, minimal, self-contained, reproducible code.