On 6/13/07, Robert Wilkins <irishhacker@gmail.com> wrote:
>
> The point is : there are lots of data preparation scenarios where
> large numbers of merges need to be done. This is an example where
> Vilno and SAS are easier to use than the competition. I'm sure an Awk
> programmer can come up with something, but the result would be
> awkward.


Agreed.
In the awk+R scenario, it is clear that the merges are often better done
with R.
My strategy is to use awk only to clean/reformat data into a tabular format
and
do most of the "consolidation" (computations/filtering/merges) in R.  I
suggested to use awk only to perform manipulations that would be more
complex to do within R (especially mutliline records or recors with
optionnal fields). I try to keep the scripts as simple as possible on both
sides



> Certain apsects of Vilno and SAS are a bit more user-friendly:
> > Each column has a variable name, such as "PatientID".
> > Awk uses $1, $2, $3 , as variable names for columns. Not user-friendly.
>
>


In the first lines of awk scripts, I usually assign column numbers to
variables (e.g. "Code=1, time=3") and then access the fields with "$Code",
"$Time"...
Yet, it is true that it is cumbersome, in awk, to use the labels on the
first line of a file as a variable names (my major complain about awk).

I looked at a few examples of  SAS Data step scripts on the Net, and found
that the awk scripts would be very similar (except for merges), but there
may  manipulations which I missed.


> For scanning inconsistently structured ASCII data files, where
> different rows have different column specifications, Awk is a better
> tool.
>
> For data problems that lend themselves to UNIX-style regular
> expressions, Awk, again, is a great tool.



The examples of messy data formats that were described ealier on the list
are good examples where regular expressions will help a lot. In the very
first stage of data inspection, to detect coding "mistakes", awk (sometimes
with the help ot other gnutools such as 'uniq' and 'sort') can be very
efficient.

> The upshot:

> Awk is a hammer.
> Vilno is a screwdriver.

Nice analogy. Using the right tool for the right task is very important.
So awk and vilno seem complementary.
Yet, when R enters into the equation, do you still "need" the three tools?

What we should really compare is the four situations:

R alone
R + awk
R + vilno
R + awk + vilno

and maybe "R + SAS Data step"

and see what scripts are more  elegant (read 'short and understandable')


Best,

Christophe



-- 
Christophe Pallier (http://www.pallier.org)

	[[alternative HTML version deleted]]

