[OGRUG] software at software-carpentry.org

[Prof. John C Nash]

On reading Greg Wilson's SIAM News item, I looked at the software-carpentry.org site and
particularly the survey report. I was quite surprised nobody responding identified as
statistician (the categories are non-exclusive, of course).

My own experience of nearly 45 years in scientific software has led me to an acute
awareness that there are islands of important work with stormy waters between. Thus the
SciPy etc. focus of software-carpentry is great for those choosing that route, but there
is also a Matlab island and, where I'm currently working, the R island, plus others. Given
the approximately 2500 add-in packages to R, it's an important focus of work, as are the
others. The big issue is that one can have a very good tool on one platform, but using it
easily on another can be very, very time-consuming and present all kinds of awkwardness.
The interface and glue software is many times the length of the core computational stuff.
I note, for example, that the course on s-c site doesn't have a graphics chapter (I see
image processing), nor does it have documentation. R has done quite a bit on making both
of these a lot easier, in particular, the Sweave tools for creating vignettes that build
documentation with code in-line. Modify the code or the data and the documentation and
graphics are updated automatically (direct to pdf). Examples and tests are part of the
framework for package building, but there is still a lot of work to do to get true unit
tests vs. test examples that are meant to show humans how to use the package.

The consequence of the islands is that we've lots of duplication and near-duplication. In
my own area of optimization (function minimization), even within R, perceived "failure" of
one package on one problem will lead people to say "you ought to use XXXX instead", when
XXXX will "fail" on many other problems. In these cases "failure" can be akin to using a
chain-saw for neurosurgery -- that is, misapplication of tools. This is a human nature
problem, and it takes a huge effort to keep the noise down to a tolerable level.

The "carpentry" metaphor is a good one. It can and should be extended to include metaphors
like the issues of interfacing different construction materials and methods and
measurement standards (metric bolts vs Imperial vs SAE).

Keep at it.

Best,

John Nash