[R] aggregate formula - differing results
Achim Zeileis
Ach|m@Ze||e|@ @end|ng |rom u|bk@@c@@t
Tue Sep 5 00:27:08 CEST 2023
On Mon, 4 Sep 2023, Ivan Calandra wrote:
> Thanks Rui for your help; that would be one possibility indeed.
>
> But am I the only one who finds that behavior of aggregate() completely
> unexpected and confusing? Especially considering that dplyr::summarise() and
> doBy::summaryBy() deal with NAs differently, even though they all use
> mean(na.rm = TRUE) to calculate the group stats.
I agree with Rui that this behaves as documented but I also agree with
Ivan that the behavior is potentially confusing. Not so much because other
packages behave differently but mostly because the handling of missing
values differs between the different aggregate() methods.
Based on my teaching experience, I feel that a default of
na.action=na.pass would be less confusing, especially in the case with
multivariate "response".
In the univeriate case the discrepancy can be surprising - in the default
method you need na.rm=TRUE but in the formula method you get the same
result without additional arguments (due to na.action=na.omit). But in the
multivariate case the discrepancy is not obvious, especially for
beginners, because the results in other variables without NAs are affected
as well.
A minimal toy example is the following data with two groups (x = A vs. B)
and two "responses" (y without NAs and z with NA):
d <- data.frame(x = c("A", "A", "B", "B"), y = 1:4, z = c(1:3, NA))
d
## x y z
## 1 A 1 1
## 2 A 2 2
## 3 B 3 3
## 4 B 4 NA
Except for naming of the columns, both of the following summaries for y by
x (without NAs) yield the same result:
aggregate(d$y, list(d$x), FUN = mean)
aggregate(y ~ x, data = d, FUN = mean)
## x y
## 1 A 1.5
## 2 B 3.5
For a single variable _with_ NAs, the default method needs the na.rm =
TRUE argument, the fomula method does not. Again, except for naming of the
columns:
aggregate(d$z, list(d$x), FUN = mean, na.rm = TRUE)
aggregate(z ~ x, data = d, FUN = mean)
## x z
## 1 A 1.5
## 2 B 3.0
Conversely, if you do want the NAs in the groups, the following two are
the same (except for naming):
aggregate(d$z, list(d$x), FUN = mean)
aggregate(z ~ x, data = d, FUN = mean, na.action = na.pass)
## x z
## 1 A 1.5
## 2 B NA
But in the multivariate case, it is not so obvious why the following two
commands differ in their results for y (!), the variable without NAs, in
group B:
aggregate(d[, c("y", "z")], list(d$x), FUN = mean, na.rm = TRUE)
## Group.1 y z
## 1 A 1.5 1.5
## 2 B 3.5 3.0
^^^
aggregate(cbind(y, z) ~ x, data = d, FUN = mean)
## x y z
## 1 A 1.5 1.5
## 2 B 3.0 3.0
^^^
Hence, in my programming courses I tell students to use na.action=na.pass
in the formula method and to handle NAs in the FUN argument themselves.
I guess that this is not important enough to change the default in
aggregate.formula. Or are there R core members who also find that this
inconsistency between the different methods is worth addressing?
If not, maybe an explicit example could be added on the help page? Showing
something like this might help:
## default: omit enitre row 4 where z=NA
aggregate(cbind(y, z) ~ x, data = d, FUN = mean)
## x y z
## 1 A 1.5 1.5
## 2 B 3.0 3.0
## alternatively: omit row 4 only for z result but not for y result
aggregate(cbind(y, z) ~ x, data = d, FUN = mean, na.action = na.pass, na.rm = TRUE)
## x y z
## 1 A 1.5 1.5
## 2 B 3.5 3.0
Best wishes,
Achim
> On 04/09/2023 13:46, Rui Barradas wrote:
>> Às 10:44 de 04/09/2023, Ivan Calandra escreveu:
>>> Dear useRs,
>>>
>>> I have just stumbled across a behavior in aggregate() that I cannot
>>> explain. Any help would be appreciated!
>>>
>>> Sample data:
>>> my_data <- structure(list(ID = c("FLINT-1", "FLINT-10", "FLINT-100",
>>> "FLINT-101", "FLINT-102", "HORN-10", "HORN-100", "HORN-102", "HORN-103",
>>> "HORN-104"), EdgeLength = c(130.75, 168.77, 142.79, 130.1, 140.41, 121.37,
>>> 70.52, 122.3, 71.01, 104.5), SurfaceArea = c(1736.87, 1571.83, 1656.46,
>>> 1247.18, 1177.47, 1169.26, 444.61, 1791.48, 461.15, 1127.2), Length =
>>> c(44.384, 29.831, 43.869, 48.011, 54.109, 41.742, 23.854, 32.075, 21.337,
>>> 35.459), Width = c(45.982, 67.303, 52.679, 26.42, 25.149, 33.427, 20.683,
>>> 62.783, 26.417, 35.297), PLATWIDTH = c(38.84, NA, 15.33, 30.37, 11.44,
>>> 14.88, 13.86, NA, NA, 26.71), PLATTHICK = c(8.67, NA, 7.99, 11.69, 3.3,
>>> 16.52, 4.58, NA, NA, 9.35), EPA = c(78, NA, 78, 54, 72, 49, 56, NA, NA,
>>> 56), THICKNESS = c(10.97, NA, 9.36, 6.4, 5.89, 11.05, 4.9, NA, NA, 10.08),
>>> WEIGHT = c(34.3, NA, 25.5, 18.6, 14.9, 29.5, 4.5, NA, NA, 23), RAWMAT =
>>> c("FLINT", "FLINT", "FLINT", "FLINT", "FLINT", "HORNFELS", "HORNFELS",
>>> "HORNFELS", "HORNFELS", "HORNFELS")), row.names = c(1L, 2L, 3L, 4L, 5L,
>>> 111L, 112L, 113L, 114L, 115L), class = "data.frame")
>>>
>>> 1) Simple aggregation with 2 variables:
>>> aggregate(cbind(Length, Width) ~ RAWMAT, data = my_data, FUN = mean, na.rm
>>> = TRUE)
>>>
>>> 2) Using the dot notation - different results:
>>> aggregate(. ~ RAWMAT, data = my_data[-1], FUN = mean, na.rm = TRUE)
>>>
>>> 3) Using dplyr, I get the same results as #1:
>>> group_by(my_data, RAWMAT) %>%
>>> summarise(across(c("Length", "Width"), ~ mean(.x, na.rm = TRUE)))
>>>
>>> 4) It gets weirder: using all columns in #1 give the same results as in #2
>>> but different from #1 and #3
>>> aggregate(cbind(EdgeLength, SurfaceArea, Length, Width, PLATWIDTH,
>>> PLATTHICK, EPA, THICKNESS, WEIGHT) ~ RAWMAT, data = my_data, FUN = mean,
>>> na.rm = TRUE)
>>>
>>> So it seems it is not only due to the notation (cbind() vs. dot). Is it a
>>> bug? A peculiar thing in my dataset? I tend to think this could be due to
>>> some variables (or their names) as all notations seem to agree when I
>>> remove some variables (although I haven't found out which variable(s) is
>>> (are) at fault), e.g.:
>>>
>>> my_data2 <- structure(list(ID = c("FLINT-1", "FLINT-10", "FLINT-100",
>>> "FLINT-101", "FLINT-102", "HORN-10", "HORN-100", "HORN-102", "HORN-103",
>>> "HORN-104"), EdgeLength = c(130.75, 168.77, 142.79, 130.1, 140.41, 121.37,
>>> 70.52, 122.3, 71.01, 104.5), SurfaceArea = c(1736.87, 1571.83, 1656.46,
>>> 1247.18, 1177.47, 1169.26, 444.61, 1791.48, 461.15, 1127.2), Length =
>>> c(44.384, 29.831, 43.869, 48.011, 54.109, 41.742, 23.854, 32.075, 21.337,
>>> 35.459), Width = c(45.982, 67.303, 52.679, 26.42, 25.149, 33.427, 20.683,
>>> 62.783, 26.417, 35.297), RAWMAT = c("FLINT", "FLINT", "FLINT", "FLINT",
>>> "FLINT", "HORNFELS", "HORNFELS", "HORNFELS", "HORNFELS", "HORNFELS")),
>>> row.names = c(1L, 2L, 3L, 4L, 5L, 111L, 112L, 113L, 114L, 115L), class =
>>> "data.frame")
>>>
>>> aggregate(cbind(EdgeLength, SurfaceArea, Length, Width) ~ RAWMAT, data =
>>> my_data2, FUN = mean, na.rm = TRUE)
>>>
>>> aggregate(. ~ RAWMAT, data = my_data2[-1], FUN = mean, na.rm = TRUE)
>>>
>>> group_by(my_data2, RAWMAT) %>%
>>> summarise(across(where(is.numeric), ~ mean(.x, na.rm = TRUE)))
>>>
>>>
>>> Thank you in advance for any hint.
>>> Best wishes,
>>> Ivan
>>>
>>>
>>>
>>>
>>> *LEIBNIZ-ZENTRUM*
>>> *FÜR ARCHÄOLOGIE*
>>>
>>> *Dr. Ivan CALANDRA*
>>> **Head of IMPALA (IMaging Platform At LeizA)
>>>
>>> *MONREPOS* Archaeological Research Centre, Schloss Monrepos
>>> 56567 Neuwied, Germany
>>>
>>> T: +49 2631 9772 243
>>> T: +49 6131 8885 543
>>> ivan.calandra using leiza.de
>>>
>>> leiza.de <http://www.leiza.de/>
>>> <http://www.leiza.de/>
>>> ORCID <https://orcid.org/0000-0003-3816-6359>
>>> ResearchGate
>>> <https://www.researchgate.net/profile/Ivan_Calandra>
>>>
>>> LEIZA is a foundation under public law of the State of
>>> Rhineland-Palatinate and the City of Mainz. Its headquarters are in Mainz.
>>> Supervision is carried out by the Ministry of Science and Health of the
>>> State of Rhineland-Palatinate. LEIZA is a research museum of the Leibniz
>>> Association.
>>> ______________________________________________
>>> R-help using r-project.org mailing list -- To UNSUBSCRIBE and more, see
>>> https://stat.ethz.ch/mailman/listinfo/r-help
>>> PLEASE do read the posting guide
>>> http://www.R-project.org/posting-guide.html
>>> and provide commented, minimal, self-contained, reproducible code.
>> Hello,
>>
>> You can define a vector of the columns of interest and subset the data with
>> it. Then the default na.action = na.omit will no longer remove the rows
>> with NA vals in at least one column and the results are the same.
>>
>> However, this will not give the mean values of the other numeric columns,
>> just of those two.
>>
>>
>>
>> # define a vector of columns of interest
>> cols <- c("Length", "Width", "RAWMAT")
>>
>> # 1) Simple aggregation with 2 variables, select cols:
>> aggregate(cbind(Length, Width) ~ RAWMAT, data = my_data[cols], FUN = mean,
>> na.rm = TRUE)
>>
>> # 2) Using the dot notation - if cols are selected, equal results:
>> aggregate(. ~ RAWMAT, data = my_data[cols], FUN = mean, na.rm = TRUE)
>>
>> # 3) Using dplyr, the results are now the same results as #1 and #2:
>> my_data %>%
>> select(all_of(cols)) %>%
>> group_by(RAWMAT) %>%
>> summarise(across(c("Length", "Width"), ~ mean(.x, na.rm = TRUE)))
>>
>>
>> Hope this helps,
>>
>> Rui Barradas
>>
>
> ______________________________________________
> R-help using r-project.org mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>
More information about the R-help
mailing list