scoping

[Luke Tierney]

Martyn Plummer wrote:
> 
> On 26-Jun-98 Paul Gilbert wrote:
> >Can Luke Tierney's recent S-new post on scoping be included in the R FAQ?
> 
> Could this be cross-posted?
> 

Here is the posting -- it was part of a thread on scoping in S.

John Maindonald wrote:
> 
> Jens Oehlschlaegel-Akiyoshi wrote, in response to John Thaden
> 
> > in S+ no function has automatic access to any local variables of any
> > parent functions FOR SECURITY REASONS (except for objects in frames 0+1). 
> There must of course be some scoping rules, and what scoping rules]
> are at the end of the day preferable is a matter for debate.

This debate has been pretty much settled in computer
science. Virtually all modern functional and near functional languages
use lexical scope. This includes all modern Lisp variants, the ML
family, Haskell, Miranda, etc. The theoretical underpinnings for these
languages is provided by the lambda calculus, which uses lexical
scope. Lexical scope is now considered essential for a higher level
functional language (i.e. a functional language that uses functions as
first-class values). This is the main reason S, even though it has
first class function values, is rarely used as a higher order
functional language. (Another reason is that most S users do not come
to S with a background of using a functional style, and with the lack
of lexical scoping it is probably not worth learning in the context of
S.)

In lexical scope, the meaning of a free variable in a function is
determined by the context in which the function is defined. For
example, in R (which Doug Bates mentioned in this thread and which is
lexically scoped)

	poisson.loglik<-function(x) {
				xbar<-mean(x)
				n<-length(x)
				function(lambda){
					n * (xbar * log(lambda) - lambda)
				}
			}
	pl<-poisson.loglik(rpois(10,2))

the value of pl is a log-likelihood function for the poisson sample
created by rpois(10,2). This was the value of the free variable x in
the inner function's definition when it was created, and this value is
locked into the function object (or closure) at creation time along
with the function definition. It does not matter what variables named
x are around when the function pl is called.

Another common scoping rule is dynamic scoping, where values of free
variables are determined by the environment in which they are
used. The log-likelihood example would not work at all with dynamic
scope since the x used would depend on what x was available each place
pl is called. Dynamic scope is what you get with macro languages and
was in effect what you had with S Version 1 where you only had macros,
not functions, available. Dynamic scoping is very occasionally useful,
but as a general rule it is much more error-prone and much less
powerful than lexical scope.

S uses a different rule: If a variable is not a local variable (a
function argument or a variable created by an <- assignment) then it
is global. This rule is much less error-prone than dynamic scoping but
also much less powerful than lexical scope. Some scripting languages
also use this rule (I think Tcl and Python do, though I haven't double
checked). There are probably several reasons why S chose to use this
rule, but the most important one is probably that lexical scope,
together with the ability to change a variable's value by assignment,
means you cannot save functions to separate files by writing out their
definitions.  Functions can be interrelated in arbitrary ways through
their defining environments, and the only easy safe way to preserve
these relationships when saving to disk is to save an entire workspace
(as R does). S's approach to persistent storage is incompatible with
lexical scope for its functions. I'm sure at the time the price of
changing the approach to storage just to accommodate lexical scope
seemed far too expensive, so the simpler scoping rule was adopted.

But the simple rule is not as expressive as lexical scope, and there
are certain things that cannot be done if only this standard scoping
rule is used; the modeling functions are one example. So S provides an
alternative: By using eval/frame access/get/assign in conjunction with
S's version of lazy evaluation (I'll call this eval for short) you can
write functions whose calls behave almost any way you want to. This is
a major difference from almost all other languages I am familiar
with. In other languages the scoping rules are absolute. In S, they
can be circumvented. This ability adds a great deal of power but it
also provides an enormous amount of rope with which to hang yourself.

Most of the time you can ignore this added power of S. If you write a
function and

	only use functions whose calls follow the standard scope rules

	don't use eval (or related things)

then the S language will guarantee that calls to your function will
follow the standard scope rules. You don't need to do anything special.

If on the other hand, you use eval, or one of the functions you call
does not follow the standard scope rules, then all bets are off. If it
is your intent to write a function that follows standard scope rules,
or if you plan to deviate from those rules in some particular way (as
the modeling functions do) then it is now your responsibility as the
programmer, instead of the language's responsibility, to make sure
things work right. This makes writing the function correctly much
harder. Therefore, if you are writing a function that follows standard
scope rules and find yourself using eval because it seems like a quick
fix, you should probably back off and see if you can avoid using
eval. In very rare cases you can't, but in almost all cases you can
and should.

There was a nice example of this issue in this list about a week
ago. Someone had written a function that called trellis.device.  The
function didn't work right even though the trellis.device call worked
right at top level. It was suggested that the poster had
miss-interpreted S's scope rules. In fact, if trellis.device were a
function following standard scope rules (as the poster had every right
to expect since the documentation does not suggest that it is not)
then the poster's function would have worked correctly. It turns out
that the author of trellis.device used eval in a way that caused a
departure from standard scope rules for calls to the function.  This
use of eval seems convenient on the surface but was not
necessary---direct manipulation of a captured ... argument list
followed by do.call, which is safe, would have done the job and made
it impossible for this kind of error to occur in the first place (but
see the **** note at the bottom). The bottom line again: If you are
using eval and you are not trying to do something that departs form
standard scope rules, you probably shouldn't.

All that said, there are cases where, given the limited expressive
power of S's standard scope rules, you really do have to use eval to
achieve certain result. The modeling functions are such a case.  The
convention adopted for the modeling functions is to use dynamic
scoping for determining values for variables in formulas if explicit
data are not provided. This is a debatable choice. Dynamic scope has
its uses, but since inadvertent variable capture is so easy to fall
into and so hard to prevent, it is probably not such a good idea.
An example: 

> x<-1:10
> y<-x+rnorm(10)
> f<-function(m) lm(m)
> g<-function(x) lm(x)

The functions f and g look equivalent, but they aren't:

> f(y~x)
Call:
lm(formula = m)
...
> g(y~x)
Error in g(y ~ x): Length of variable 2 is 3 != length of row names (10)

The local variable x in g, the formula, shadows the global one and
causes lm to fail. Of course if the formula contained a variable m
then f would be in trouble. I have discussed this issue with several
individuals close to the S developers who argued that this behavior is
a feature. I do not think it is the best design choice. But it is the
design choice in use, and one can live with it if this is understood.
But since eval is needed to accomplish this behavior it is up to the
programmer to insure that this design is adhered to, rather than being
guaranteed by language design features.

A better design choice, in my view, would have been to insure that
formulas are always interpreted in the context in which they are
created, so that lm would automatically see the global x and y as the
values relevant to the formula even though it is called inside f or g
in these examples. Put another way, instead of thinking of ~ as merely
a quoting operation like it is now, it should be viewed as a
closure-creating operation that follows lexical scope rules. The
resulting formula should contain not only the expression but also (at
least conceptually) the current values of the variables referenced in
the expression, if there are any available. This approach would
produce results closer to what most users would tend to expect until
they read the fine print (the example above would work properly). It
would also eliminate the need to use eval in this context, thus
eliminating a serious source of possible programming errors.

But of course this approach comes at a price: it is essentially
introducing lexical scope, with all the old issues of separate data
storage. The design choice made in S was clearly the easier one to
implement. On the other hand, a language like R that already has
lexical scope could implement this approach quite easily, but at the
cost of losing compatibility with S.

(The idea described by Rod Ball of adjusting the frame to use for
variable lookup, if I understand it correctly, is intended to achieve
similar results. But it has two drawbacks. One, it would be very
expensive in runtime overhead, but more importantly it would only work
within the dynamic extent of the call. If a fit is created in a nested
call, stored, and then reused later in an unrelated call, things will
no longer work right. Proper lexical scope avoids this.)

(**** Actually things are not quite so simple with do.call. What the
trellis.device function needed is the ability to construct an argument
list at run time, obtain a function at run time, and then call the
function with the argument list. This is almost what do.call does,
except it expects its function argument to be a string naming the
function, not an expression whose value is a function. This seems like
a strange design, but there is probably a good reason for it. It is possible
to use the current do.call by using something like

	tmpfun<- ... expression making function ...
   	do.call("tmpfun", args)

but this is awkward. It would be cleaner to define a version of
do.call, say do.call.expr, that allows its function to be specified as
an expression. This does of course require the use of eval again, as
does the definition of do.call, but it can be done carefully once and
for all to produce a function that obeys standard scope rules, as
do.call does, and then this function can be used safely whenever this
type of operation needs to be performed. If and when there is a real
need to use eval this is a good strategy: abstract out the core that
needs eval into function, and make the function use standard scoping
or carefully document how and why it doesn't. Then you can always use
this abstraction when you need it instead of manually inserting the
corresponding code with the associated risks of errors.)

luke

-- 
Luke Tierney
University of Minnesota                      Phone:           612-625-7843
School of Statistics                         Fax:             612-624-8868
206 Church Street                            email:      luke@stat.umn.edu
Minneapolis, MN 55455 USA                    WWW:  http://www.stat.umn.edu
-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-
r-devel mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html
Send "info", "help", or "[un]subscribe"
(in the "body", not the subject !)  To: r-devel-request@stat.math.ethz.ch
_._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._