[R] pairwise deletion in regression models

[Fox, John]

Dear Petr and Adel,

I'm unaware of any R software that does "pairwise deletion" of missing data (that is, uses pairwise-complete cases). It's possible to compute LS regression from covariances and means (or, equivalently, correlations, standard deviations, and means), which makes this approach possible, and it would be very simple to program it in R. 

Generally, however, pairwise deletion of missing data is a poor approach, providing consistent estimates of the regression coefficients only when missing data are missing completely at random (MCAR). Complete-case analysis ("listwise deletion" of missing data in SPSS jargon), on the other hand, also provides consistent estimates when data are missing at random (MAR) and missingness doesn't depend on the response variable. It can also, counterintuitively, be more efficient then pairwise deletion of missing data. As I recall, there's a nice example of this in Little and Rubin, Statistical Analysis of Missing Data, 2nd Ed. (Wiley, 2002), which is a good, accessible treatment of the topic. I have some notes on the subject from a course at <http://socserv.socsci.mcmaster.ca/jfox/Courses/soc740/Missing-data-notes.pdf>.

There are some other deficiencies of pairwise deletion of missing data that I won't go into here.

So, it would be easy to do pairwise deletion of missing data in R -- just write a simple program -- but I wouldn't generally advise it. If you have just a little missing data, you can do a complete-case analysis. Otherwise, there are better approaches, such as multiple imputation of missing values, for which there are many implementations in R -- e.g., the mi, mice, and amelia package.

I hope this helps,
 John

-----------------------------
John Fox, Professor
McMaster University
Hamilton, Ontario
Canada L8S 4M4
Web: socserv.mcmaster.ca/jfox


> -----Original Message-----
> From: R-help [mailto:r-help-bounces at r-project.org] On Behalf Of PIKAL Petr
> Sent: July 13, 2016 5:14 AM
> To: Adel Daoud <adel.daoud at sociology.gu.se>
> Cc: r-help <r-help at r-project.org>
> Subject: Re: [R] pairwise deletion in regression models
> 
> Hi
> 
> maybe others can give you definitive answer
> 
> see my comments in line
> 
> From: adeldaoud at gmail.com [mailto:adeldaoud at gmail.com] On Behalf Of
> Adel Daoud
> Sent: Wednesday, July 13, 2016 1:42 PM
> To: PIKAL Petr <petr.pikal at precheza.cz>
> Cc: r-help <r-help at r-project.org>
> Subject: Re: [R] pairwise deletion in regression models
> 
> Hi
> Yes, I am sure. Have a look here for SPSS e.g.:
> http://www-01.ibm.com/support/docview.wss?uid=swg21475199 and here
> http://www.ats.ucla.edu/stat/spss/modules/missing.htm
> Well there is no explicite statement that pairwise deletion of missing values is
> performed. However I admit that this REGRESSION If values of any of the
> variables on the var subcommand are missing, the entire case is excluded from
> the analysis (i.e., listwise deletion of missing data). It is possible to further
> control the treatment of missing data with the missing subcommand and one
> of the following keywords: pairwise, meansubstitution, or include.
> states that there is pairwise option in regression missing values treatment and
> for STATA here, http://www.ats.ucla.edu/stat/stata/modules/missing.html
> Seems to me that Stata performs listwise missing reg If any of the variables
> listed after the reg command are missing, the observations missing that
> value(s) are excluded from the analysis (i.e., listwise deletion of missing data).
> Cheers
> Petr
> 
> I know that R allows for pairwise deletion in the cor() function, but I need it for
> regression analysis. The default is listwise (casewise) deletion. Would be
> grateful for further input on this.
> 
> 
> 
> 
> Best,
> 
> Adel
> 
> --
> 
> Adel Daoud, PhD, Researcher
> 
> Newly published:
> 
> * Daoud, Adel and Kohl, Sebastian, How Much Do Sociologists Write About
> Economic Topics? Using Big-Data to Test Some Conventional Views in Economic
> Sociology, 1890 to 2014. Max Planck Institute for the Study of Societies,
> Discussion Paper 16/7 <http://www.mpifg.de/pu/mpifg_dp/dp16-7.pdf>
> 
> * Daoud, Adel, Björn Halleröd, and Debarati Guha-Sapir, (2016) “What Is the
> Association between Absolute Child Poverty, Poor Governance, and Natural
> Disasters? A Global Comparison of Some of the Realities of Climate Change”,
> PLoS ONE
> 11(4)<http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0153
> 296>
> 
> *Shailen Nandy, Adel Daoud, David Gordon, (2016), Examining the changing
> profile of undernutrition in the context of food price rises and greater
> inequality, Social Science & Medicine, Volume 149, Pages 153–
> 163<http://www.sciencedirect.com/science/article/pii/S0277953615302446>
> 
> 
> 
> Department of Sociology and Work Science,
> 
> University of Gothenburg
> 
> Box 720
> 
> 405 30, Göteborg, Sweden
> 
> Email: Adel.daoud at sociology.gu.se<mailto:Adel.daoud at sociology.gu.se>
> 
> Website: http://adeldaoud.se/ <http://adeldaoud.se/>
> 
> On Wed, Jul 13, 2016 at 10:53 AM, PIKAL Petr
> <petr.pikal at precheza.cz<mailto:petr.pikal at precheza.cz>> wrote:
> Hi
> 
> Hm. Are you 100% sure that other software packages can do pairwise deletion
> in OLS or GLM? I am not at all familiar with them but
> 
> http://www.ats.ucla.edu/stat/spss/modules/stats.htm
> 
> suggests that option pairwise is available with corr and I believe the same
> option exists in cor function in R.
> 
> My statistical knowledge is inferior but I just cannot imagine how whole model
> could be computed when one value is missing.
> 
> Cheers
> Petr
> 
> From: adeldaoud at gmail.com<mailto:adeldaoud at gmail.com>
> [mailto:adeldaoud at gmail.com<mailto:adeldaoud at gmail.com>] On Behalf Of
> Adel Daoud
> Sent: Wednesday, July 13, 2016 10:00 AM
> To: PIKAL Petr <petr.pikal at precheza.cz<mailto:petr.pikal at precheza.cz>>
> Cc: r-help <r-help at r-project.org<mailto:r-help at r-project.org>>
> Subject: Re: [R] pairwise deletion in regression models
> 
> Thanks Petr for the suggestion.
> 
> I just took the regtools package for a quick test drive. It looks promising, but it
> still needs further development to make it a viable option. You will not get a
> standard regression output (as in lm or glm), only the regression coefficients
> (without standard errors). I will be happy to try it out once it is more robust.
> 
> I am a bit puzzled that the R universe seems to lack a robust package that
> allows for pairwise deletion (which is standard in the otherwise poorer
> software packages STATA or SPSS…). Would be very happy if anyone can show
> me otherwise.
> 
> 
> Best,
> 
> Adel
> 
> --
> 
> Adel Daoud, PhD, Researcher
> 
> Newly published:
> 
> * Daoud, Adel and Kohl, Sebastian, How Much Do Sociologists Write About
> Economic Topics? Using Big-Data to Test Some Conventional Views in Economic
> Sociology, 1890 to 2014. Max Planck Institute for the Study of Societies,
> Discussion Paper 16/7 <http://www.mpifg.de/pu/mpifg_dp/dp16-7.pdf>
> 
> * Daoud, Adel, Björn Halleröd, and Debarati Guha-Sapir, (2016) “What Is the
> Association between Absolute Child Poverty, Poor Governance, and Natural
> Disasters? A Global Comparison of Some of the Realities of Climate Change”,
> PLoS ONE
> 11(4)<http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0153
> 296>
> 
> *Shailen Nandy, Adel Daoud, David Gordon, (2016), Examining the changing
> profile of undernutrition in the context of food price rises and greater
> inequality, Social Science & Medicine, Volume 149, Pages 153–
> 163<http://www.sciencedirect.com/science/article/pii/S0277953615302446>
> 
> 
> 
> Department of Sociology and Work Science,
> 
> University of Gothenburg
> 
> Box 720
> 
> 405 30, Göteborg, Sweden
> 
> Email: Adel.daoud at sociology.gu.se<mailto:Adel.daoud at sociology.gu.se>
> 
> Website: http://adeldaoud.se/ <http://adeldaoud.se/>
> 
> On Wed, Jul 13, 2016 at 8:21 AM, PIKAL Petr
> <petr.pikal at precheza.cz<mailto:petr.pikal at precheza.cz>> wrote:
> Hi
> 
> http://stats.stackexchange.com/questions/158366/fit-multiple-regression-
> model-with-pairwise-deletion-or-on-a-correlation-covari
> 
> The package is probably not available on CRAN but seems to be still maintained
> on github.
> 
> Cheers
> Petr
> 
> > -----Original Message-----
> > From: R-help
> > [mailto:r-help-bounces at r-project.org<mailto:r-help-bounces at r-project.o
> > rg>] On Behalf Of adel daoud
> > Sent: Tuesday, July 12, 2016 8:28 PM
> > To: r-help <r-help at r-project.org<mailto:r-help at r-project.org>>
> > Subject: [R] pairwise deletion in regression models
> >
> > Dear R users,
> >
> >
> >
> > I would like to use a pairwise deletion of missing values in linear
> > regression (lm or glm preferably). I want to replicate some studies
> > done in STATA that uses this type of deletion. What options do we have
> > in R to work with pairwise deletion? Most packages I have found do not
> > have this option, it seems (lm, glm, plm, psych, sampleSelection).
> >
> >
> >
> > This question has been raised here
> > <http://r.789695.n4.nabble.com/set-the-bahavior-that-R-deal-with-missi
> > ng-
> > values-td803840.html>
> > and here
> > <http://r.789695.n4.nabble.com/Pairwise-deletion-in-a-linear-regressio
> > n-
> > and-in-a-GLM-td4653004.html>,
> > but without any clear answer.
> >
> >
> >
> > Any input is welcomed
> >
> >
> >
> > Thanks in advance
> >
> >
> > Adel
> >
> >       [[alternative HTML version deleted]]
> >
> > ______________________________________________
> > R-help at r-project.org<mailto:R-help at r-project.org> mailing list -- To
> > UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help
> > PLEASE do read the posting guide http://www.R-project.org/posting-
> > guide.html and provide commented, minimal, self-contained,
> > reproducible code.
> 
> ________________________________
> Tento e-mail a jakékoliv k němu připojené dokumenty jsou důvěrné a jsou
> určeny pouze jeho adresátům.
> Jestliže jste obdržel(a) tento e-mail omylem, informujte laskavě neprodleně
> jeho odesílatele. Obsah tohoto emailu i s přílohami a jeho kopie vymažte ze
> svého systému.
> Nejste-li zamýšleným adresátem tohoto emailu, nejste oprávněni tento email
> jakkoliv užívat, rozšiřovat, kopírovat či zveřejňovat.
> Odesílatel e-mailu neodpovídá za eventuální škodu způsobenou modifikacemi
> či zpožděním přenosu e-mailu.
> 
> V případě, že je tento e-mail součástí obchodního jednání:
> - vyhrazuje si odesílatel právo ukončit kdykoliv jednání o uzavření smlouvy, a to
> z jakéhokoliv důvodu i bez uvedení důvodu.
> - a obsahuje-li nabídku, je adresát oprávněn nabídku bezodkladně přijmout;
> Odesílatel tohoto e-mailu (nabídky) vylučuje přijetí nabídky ze strany příjemce
> s dodatkem či odchylkou.
> - trvá odesílatel na tom, že příslušná smlouva je uzavřena teprve výslovným
> dosažením shody na všech jejích náležitostech.
> - odesílatel tohoto emailu informuje, že není oprávněn uzavírat za společnost
> žádné smlouvy s výjimkou případů, kdy k tomu byl písemně zmocněn nebo
> písemně pověřen a takové pověření nebo plná moc byly adresátovi tohoto
> emailu případně osobě, kterou adresát zastupuje, předloženy nebo jejich
> existence je adresátovi či osobě jím zastoupené známá.
> 
> This e-mail and any documents attached to it may be confidential and are
> intended only for its intended recipients.
> If you received this e-mail by mistake, please immediately inform its sender.
> Delete the contents of this e-mail with all attachments and its copies from your
> system.
> If you are not the intended recipient of this e-mail, you are not authorized to
> use, disseminate, copy or disclose this e-mail in any manner.
> The sender of this e-mail shall not be liable for any possible damage caused by
> modifications of the e-mail or by delay with transfer of the email.
> 
> In case that this e-mail forms part of business dealings:
> - the sender reserves the right to end negotiations about entering into a
> contract in any time, for any reason, and without stating any reasoning.
> - if the e-mail contains an offer, the recipient is entitled to immediately accept
> such offer; The sender of this e-mail (offer) excludes any acceptance of the
> offer on the part of the recipient containing any amendment or variation.
> - the sender insists on that the respective contract is concluded only upon an
> express mutual agreement on all its aspects.
> - the sender of this e-mail informs that he/she is not authorized to enter into
> any contracts on behalf of the company except for cases in which he/she is
> expressly authorized to do so in writing, and such authorization or power of
> attorney is submitted to the recipient or the person represented by the
> recipient, or the existence of such authorization is known to the recipient of the
> person represented by the recipient.
> 
> 
> ________________________________
> Tento e-mail a jakékoliv k němu připojené dokumenty jsou důvěrné a jsou
> určeny pouze jeho adresátům.
> Jestliže jste obdržel(a) tento e-mail omylem, informujte laskavě neprodleně
> jeho odesílatele. Obsah tohoto emailu i s přílohami a jeho kopie vymažte ze
> svého systému.
> Nejste-li zamýšleným adresátem tohoto emailu, nejste oprávněni tento email
> jakkoliv užívat, rozšiřovat, kopírovat či zveřejňovat.
> Odesílatel e-mailu neodpovídá za eventuální škodu způsobenou modifikacemi
> či zpožděním přenosu e-mailu.
> 
> V případě, že je tento e-mail součástí obchodního jednání:
> - vyhrazuje si odesílatel právo ukončit kdykoliv jednání o uzavření smlouvy, a to
> z jakéhokoliv důvodu i bez uvedení důvodu.
> - a obsahuje-li nabídku, je adresát oprávněn nabídku bezodkladně přijmout;
> Odesílatel tohoto e-mailu (nabídky) vylučuje přijetí nabídky ze strany příjemce
> s dodatkem či odchylkou.
> - trvá odesílatel na tom, že příslušná smlouva je uzavřena teprve výslovným
> dosažením shody na všech jejích náležitostech.
> - odesílatel tohoto emailu informuje, že není oprávněn uzavírat za společnost
> žádné smlouvy s výjimkou případů, kdy k tomu byl písemně zmocněn nebo
> písemně pověřen a takové pověření nebo plná moc byly adresátovi tohoto
> emailu případně osobě, kterou adresát zastupuje, předloženy nebo jejich
> existence je adresátovi či osobě jím zastoupené známá.
> 
> This e-mail and any documents attached to it may be confidential and are
> intended only for its intended recipients.
> If you received this e-mail by mistake, please immediately inform its sender.
> Delete the contents of this e-mail with all attachments and its copies from your
> system.
> If you are not the intended recipient of this e-mail, you are not authorized to
> use, disseminate, copy or disclose this e-mail in any manner.
> The sender of this e-mail shall not be liable for any possible damage caused by
> modifications of the e-mail or by delay with transfer of the email.
> 
> In case that this e-mail forms part of business dealings:
> - the sender reserves the right to end negotiations about entering into a
> contract in any time, for any reason, and without stating any reasoning.
> - if the e-mail contains an offer, the recipient is entitled to immediately accept
> such offer; The sender of this e-mail (offer) excludes any acceptance of the
> offer on the part of the recipient containing any amendment or variation.
> - the sender insists on that the respective contract is concluded only upon an
> express mutual agreement on all its aspects.
> - the sender of this e-mail informs that he/she is not authorized to enter into
> any contracts on behalf of the company except for cases in which he/she is
> expressly authorized to do so in writing, and such authorization or power of
> attorney is submitted to the recipient or the person represented by the
> recipient, or the existence of such authorization is known to the recipient of the
> person represented by the recipient.
> 
> 
> ________________________________
> Tento e-mail a jakékoliv k němu připojené dokumenty jsou důvěrné a jsou
> určeny pouze jeho adresátům.
> Jestliže jste obdržel(a) tento e-mail omylem, informujte laskavě neprodleně
> jeho odesílatele. Obsah tohoto emailu i s přílohami a jeho kopie vymažte ze
> svého systému.
> Nejste-li zamýšleným adresátem tohoto emailu, nejste oprávněni tento email
> jakkoliv užívat, rozšiřovat, kopírovat či zveřejňovat.
> Odesílatel e-mailu neodpovídá za eventuální škodu způsobenou modifikacemi
> či zpožděním přenosu e-mailu.
> 
> V případě, že je tento e-mail součástí obchodního jednání:
> - vyhrazuje si odesílatel právo ukončit kdykoliv jednání o uzavření smlouvy, a to
> z jakéhokoliv důvodu i bez uvedení důvodu.
> - a obsahuje-li nabídku, je adresát oprávněn nabídku bezodkladně přijmout;
> Odesílatel tohoto e-mailu (nabídky) vylučuje přijetí nabídky ze strany příjemce
> s dodatkem či odchylkou.
> - trvá odesílatel na tom, že příslušná smlouva je uzavřena teprve výslovným
> dosažením shody na všech jejích náležitostech.
> - odesílatel tohoto emailu informuje, že není oprávněn uzavírat za společnost
> žádné smlouvy s výjimkou případů, kdy k tomu byl písemně zmocněn nebo
> písemně pověřen a takové pověření nebo plná moc byly adresátovi tohoto
> emailu případně osobě, kterou adresát zastupuje, předloženy nebo jejich
> existence je adresátovi či osobě jím zastoupené známá.
> 
> This e-mail and any documents attached to it may be confidential and are
> intended only for its intended recipients.
> If you received this e-mail by mistake, please immediately inform its sender.
> Delete the contents of this e-mail with all attachments and its copies from your
> system.
> If you are not the intended recipient of this e-mail, you are not authorized to
> use, disseminate, copy or disclose this e-mail in any manner.
> The sender of this e-mail shall not be liable for any possible damage caused by
> modifications of the e-mail or by delay with transfer of the email.
> 
> In case that this e-mail forms part of business dealings:
> - the sender reserves the right to end negotiations about entering into a
> contract in any time, for any reason, and without stating any reasoning.
> - if the e-mail contains an offer, the recipient is entitled to immediately accept
> such offer; The sender of this e-mail (offer) excludes any acceptance of the
> offer on the part of the recipient containing any amendment or variation.
> - the sender insists on that the respective contract is concluded only upon an
> express mutual agreement on all its aspects.
> - the sender of this e-mail informs that he/she is not authorized to enter into
> any contracts on behalf of the company except for cases in which he/she is
> expressly authorized to do so in writing, and such authorization or power of
> attorney is submitted to the recipient or the person represented by the
> recipient, or the existence of such authorization is known to the recipient of the
> person represented by the recipient.
> 
> 	[[alternative HTML version deleted]]
> 
> ______________________________________________
> R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-
> guide.html
> and provide commented, minimal, self-contained, reproducible code.