[R] a fast way to do my job

[Yuan Chun Ding]

HI Bert and Ben,

Yes, running lm.fit using the matrix format is much faster. I read a couple of online comments why it is faster.

However, the residual values for three tested variables or genes from lm function and lm.fit function are different, with Pearson correlation of 0.55, 0.89, and 0.99.

I have not found the reason.

Thanks,

Ding

From: Bert Gunter <bgunter.4567 using gmail.com>
Sent: Friday, August 9, 2024 7:11 PM
To: Ben Bolker <bbolker using gmail.com>
Cc: Yuan Chun Ding <ycding using coh.org>; r-help using r-project.org
Subject: Re: [R] a fast way to do my job

Better idea, Ben! It would work as you might expect it to to produce the same results as the above: ##first make sure your regressor is a matrix: pur2 <- matrix(purity2, ncol =1) ## convert the data frame variables into a matrix dat <-


Better idea, Ben!



It would work as you might expect it to to produce the same results as

the above:



##first make sure your regressor is a matrix:

pur2 <- matrix(purity2, ncol =1)

## convert the data frame variables into a matrix

dat <- as.matrix(gem751be.rpkm[ , 74:35164])

##then

result <- residuals(lm.fit( x= pur2, y = dat))



Cheers,

Bert



On Fri, Aug 9, 2024 at 6:38 PM Ben Bolker <bbolker using gmail.com<mailto:bbolker using gmail.com>> wrote:

>

> You can also fit a linear model with a matrix-valued response

> variable, which should be even faster (not sure off the top of my head

> how to get the residuals and reshape them to the dimensions you want)

>

> On Fri, Aug 9, 2024 at 9:31 PM Bert Gunter <bgunter.4567 using gmail.com<mailto:bgunter.4567 using gmail.com>> wrote:

> >

> > See ?lm.fit.

> > I must be missing something, because:

> >

> > results <- sapply(74:35164, \(i) residuals(lm.fit(purity2,

> > gem751be.rpkm[, i] )))

> >

> > would give you a 751 x 35091 matrix of the residuals from each of the

> > regressions.

> > I assume it will be considerably faster than all the overhead you are

> > carrying in your current code, but of course you'll have to try it and

> > see. ... Assuming that I have interpreted your request correctly.

> > Ignore if not.

> >

> > Cheers,

> > Bert

> >

> > On Fri, Aug 9, 2024 at 4:50 PM Yuan Chun Ding via R-help

> > <r-help using r-project.org<mailto:r-help using r-project.org>> wrote:

> > >

> > > Dear R users,

> > >

> > > I am running the following code below,  the gem751be.rpkm is a dataframe with dim of 751 samples by 35164 variables,  73 phenotypic variables in the furst to 73rd column and 35091 genomic variables or genes in the 74th to 35164th columns.  What I need to do is to calculate the residuals for each gene using the simple linear regression model of genelist[i] ~ purity2;

> > >

> > > The following code is running,  it takes long time, but I have an expensive ThinkStation window computer.

> > > Can you provide a fast way to do it?

> > >

> > > Thank you,

> > >

> > > Ding

> > >

> > > ---------------------------------------------------------------------------------

> > >

> > >

> > > gem751be.rpkm <-merge(gem751be10, as.data.frame(t(rna849.fpkm2)),

> > > +                           by.x="id2",by.y=0)

> > > >   row.names(gem751be.rpkm)<-gem751be.rpkm$id3

> > > >   colnames(gem751be.rpkm)<-gsub(colnames(gem751be.rpkm),pattern="-",replacement="_")

> > > >   genelist <- gem751be.rpkm %>% dplyr::select(74:35164)

> > > >   residuals <- NULL

> > > >   for (i in 1:length(genelist)) {

> > > +     #i=1

> > > +     formula <- reformulate("purity2", response=names(genelist)[i])

> > > +     model <- lm(formula, data = gem751be.rpkm)

> > > +     resi <- as.data.frame(residuals(model))

> > > +     colnames(resi)[1]<-names(genelist)[i]

> > > +     resi <-as.data.frame(t(resi))

> > > +     residuals <- rbind(residuals, resi)

> > > +   }

> > >

> > >

> > >

> > > ----------------------------------------------------------------------

> > > ------------------------------------------------------------

> > > -SECURITY/CONFIDENTIALITY WARNING-

> > >

> > > This message and any attachments are intended solely for the individual or entity to which they are addressed. This communication may contain information that is privileged, confidential, or exempt from disclosure under applicable law (e.g., personal health information, research data, financial information). Because this e-mail has been sent without encryption, individuals other than the intended recipient may be able to view the information, forward it to others or tamper with the information without the knowledge or consent of the sender. If you are not the intended recipient, or the employee or person responsible for delivering the message to the intended recipient, any dissemination, distribution or copying of the communication is strictly prohibited. If you received the communication in error, please notify the sender immediately by replying to this message and deleting the message and any accompanying files from your system. If, due to the security risks, you do not wish to rec

> > >  eive further communications via e-mail, please reply to this message and inform the sender that you do not wish to receive further e-mail from the sender. (LCP301)

> > > ------------------------------------------------------------

> > >

> > >         [[alternative HTML version deleted]]

> > >

> > > ______________________________________________

> > > R-help using r-project.org<mailto:R-help using r-project.org> mailing list -- To UNSUBSCRIBE and more, see

> > > https://urldefense.com/v3/__https://stat.ethz.ch/mailman/listinfo/r-help__;!!Fou38LsQmgU!qcJ4z-vlMNzsa8XCsJUcuPOz8Vt12zsV_XaWpqXsyUYJBTlcNRonFPr7w7Ql3xqcDnZ9ZYC8JX72PW30DQ$<https://urldefense.com/v3/__https:/stat.ethz.ch/mailman/listinfo/r-help__;!!Fou38LsQmgU!qcJ4z-vlMNzsa8XCsJUcuPOz8Vt12zsV_XaWpqXsyUYJBTlcNRonFPr7w7Ql3xqcDnZ9ZYC8JX72PW30DQ$>

> > > PLEASE do read the posting guide https://urldefense.com/v3/__http://www.R-project.org/posting-guide.html__;!!Fou38LsQmgU!qcJ4z-vlMNzsa8XCsJUcuPOz8Vt12zsV_XaWpqXsyUYJBTlcNRonFPr7w7Ql3xqcDnZ9ZYC8JX66rfmKvA$<https://urldefense.com/v3/__http:/www.R-project.org/posting-guide.html__;!!Fou38LsQmgU!qcJ4z-vlMNzsa8XCsJUcuPOz8Vt12zsV_XaWpqXsyUYJBTlcNRonFPr7w7Ql3xqcDnZ9ZYC8JX66rfmKvA$>

> > > and provide commented, minimal, self-contained, reproducible code.

> >

> > ______________________________________________

> > R-help using r-project.org<mailto:R-help using r-project.org> mailing list -- To UNSUBSCRIBE and more, see

> > https://urldefense.com/v3/__https://stat.ethz.ch/mailman/listinfo/r-help__;!!Fou38LsQmgU!qcJ4z-vlMNzsa8XCsJUcuPOz8Vt12zsV_XaWpqXsyUYJBTlcNRonFPr7w7Ql3xqcDnZ9ZYC8JX72PW30DQ$<https://urldefense.com/v3/__https:/stat.ethz.ch/mailman/listinfo/r-help__;!!Fou38LsQmgU!qcJ4z-vlMNzsa8XCsJUcuPOz8Vt12zsV_XaWpqXsyUYJBTlcNRonFPr7w7Ql3xqcDnZ9ZYC8JX72PW30DQ$>

> > PLEASE do read the posting guide https://urldefense.com/v3/__http://www.R-project.org/posting-guide.html__;!!Fou38LsQmgU!qcJ4z-vlMNzsa8XCsJUcuPOz8Vt12zsV_XaWpqXsyUYJBTlcNRonFPr7w7Ql3xqcDnZ9ZYC8JX66rfmKvA$<https://urldefense.com/v3/__http:/www.R-project.org/posting-guide.html__;!!Fou38LsQmgU!qcJ4z-vlMNzsa8XCsJUcuPOz8Vt12zsV_XaWpqXsyUYJBTlcNRonFPr7w7Ql3xqcDnZ9ZYC8JX66rfmKvA$>

> > and provide commented, minimal, self-contained, reproducible code.

	[[alternative HTML version deleted]]