[R] a fast way to do my job

Sat Aug 10 22:02:51 CEST 2024

Is it because I failed to to add a column of ones for an intercept to
the x matrix? TRhat would be my bad.

-- Bert

On Sat, Aug 10, 2024 at 12:59 PM Bert Gunter <bgunter.4567 using gmail.com> wrote:
>
> Probably because you inadvertently ran different models. Without your code, I haven't a clue.
>
>
> On Sat, Aug 10, 2024, 12:29 Yuan Chun Ding <ycding using coh.org> wrote:
>>
>> HI Bert and Ben,
>>
>>
>>
>> Yes, running lm.fit using the matrix format is much faster. I read a couple of online comments why it is faster.
>>
>>
>>
>> However, the residual values for three tested variables or genes from lm function and lm.fit function are different, with Pearson correlation of 0.55, 0.89, and 0.99.
>>
>>
>>
>> I have not found the reason.
>>
>>
>>
>> Thanks,
>>
>>
>> Ding
>>
>>
>>
>> From: Bert Gunter <bgunter.4567 using gmail.com>
>> Sent: Friday, August 9, 2024 7:11 PM
>> To: Ben Bolker <bbolker using gmail.com>
>> Cc: Yuan Chun Ding <ycding using coh.org>; r-help using r-project.org
>> Subject: Re: [R] a fast way to do my job
>>
>>
>>
>> Better idea, Ben! It would work as you might expect it to to produce the same results as the above: ##first make sure your regressor is a matrix: pur2 <- matrix(purity2, ncol =1) ## convert the data frame variables into a matrix dat <-
>>
>> Better idea, Ben!
>>
>>
>>
>> It would work as you might expect it to to produce the same results as
>>
>> the above:
>>
>>
>>
>> ##first make sure your regressor is a matrix:
>>
>> pur2 <- matrix(purity2, ncol =1)
>>
>> ## convert the data frame variables into a matrix
>>
>> dat <- as.matrix(gem751be.rpkm[ , 74:35164])
>>
>> ##then
>>
>> result <- residuals(lm.fit( x= pur2, y = dat))
>>
>>
>>
>> Cheers,
>>
>> Bert
>>
>>
>>
>> On Fri, Aug 9, 2024 at 6:38 PM Ben Bolker <bbolker using gmail.com> wrote:
>>
>> >
>>
>> > You can also fit a linear model with a matrix-valued response
>>
>> > variable, which should be even faster (not sure off the top of my head
>>
>> > how to get the residuals and reshape them to the dimensions you want)
>>
>> >
>>
>> > On Fri, Aug 9, 2024 at 9:31 PM Bert Gunter <bgunter.4567 using gmail.com> wrote:
>>
>> > >
>>
>> > > See ?lm.fit.
>>
>> > > I must be missing something, because:
>>
>> > >
>>
>> > > results <- sapply(74:35164, \(i) residuals(lm.fit(purity2,
>>
>> > > gem751be.rpkm[, i] )))
>>
>> > >
>>
>> > > would give you a 751 x 35091 matrix of the residuals from each of the
>>
>> > > regressions.
>>
>> > > I assume it will be considerably faster than all the overhead you are
>>
>> > > carrying in your current code, but of course you'll have to try it and
>>
>> > > see. ... Assuming that I have interpreted your request correctly.
>>
>> > > Ignore if not.
>>
>> > >
>>
>> > > Cheers,
>>
>> > > Bert
>>
>> > >
>>
>> > > On Fri, Aug 9, 2024 at 4:50 PM Yuan Chun Ding via R-help
>>
>> > > <r-help using r-project.org> wrote:
>>
>> > > >
>>
>> > > > Dear R users,
>>
>> > > >
>>
>> > > > I am running the following code below,  the gem751be.rpkm is a dataframe with dim of 751 samples by 35164 variables,  73 phenotypic variables in the furst to 73rd column and 35091 genomic variables or genes in the 74th to 35164th columns.  What I need to do is to calculate the residuals for each gene using the simple linear regression model of genelist[i] ~ purity2;
>>
>> > > >
>>
>> > > > The following code is running,  it takes long time, but I have an expensive ThinkStation window computer.
>>
>> > > > Can you provide a fast way to do it?
>>
>> > > >
>>
>> > > > Thank you,
>>
>> > > >
>>
>> > > > Ding
>>
>> > > >
>>
>> > > > ---------------------------------------------------------------------------------
>>
>> > > >
>>
>> > > >
>>
>> > > > gem751be.rpkm <-merge(gem751be10, as.data.frame(t(rna849.fpkm2)),
>>
>> > > > +                           by.x="id2",by.y=0)
>>
>> > > > >   row.names(gem751be.rpkm)<-gem751be.rpkm$id3
>>
>> > > > >   colnames(gem751be.rpkm)<-gsub(colnames(gem751be.rpkm),pattern="-",replacement="_")
>>
>> > > > >   genelist <- gem751be.rpkm %>% dplyr::select(74:35164)
>>
>> > > > >   residuals <- NULL
>>
>> > > > >   for (i in 1:length(genelist)) {
>>
>> > > > +     #i=1
>>
>> > > > +     formula <- reformulate("purity2", response=names(genelist)[i])
>>
>> > > > +     model <- lm(formula, data = gem751be.rpkm)
>>
>> > > > +     resi <- as.data.frame(residuals(model))
>>
>> > > > +     colnames(resi)[1]<-names(genelist)[i]
>>
>> > > > +     resi <-as.data.frame(t(resi))
>>
>> > > > +     residuals <- rbind(residuals, resi)
>>
>> > > > +   }
>>
>> > > >
>>
>> > > >
>>
>> > > >
>>
>> > > > ----------------------------------------------------------------------
>>
>> > > > ------------------------------------------------------------
>>
>> > > > -SECURITY/CONFIDENTIALITY WARNING-
>>
>> > > >
>>
>> > > > This message and any attachments are intended solely for the individual or entity to which they are addressed. This communication may contain information that is privileged, confidential, or exempt from disclosure under applicable law (e.g., personal health information, research data, financial information). Because this e-mail has been sent without encryption, individuals other than the intended recipient may be able to view the information, forward it to others or tamper with the information without the knowledge or consent of the sender. If you are not the intended recipient, or the employee or person responsible for delivering the message to the intended recipient, any dissemination, distribution or copying of the communication is strictly prohibited. If you received the communication in error, please notify the sender immediately by replying to this message and deleting the message and any accompanying files from your system. If, due to the security risks, you do not wish to rec
>>
>> > > >  eive further communications via e-mail, please reply to this message and inform the sender that you do not wish to receive further e-mail from the sender. (LCP301)
>>
>> > > > ------------------------------------------------------------
>>
>> > > >
>>
>> > > >         [[alternative HTML version deleted]]
>>
>> > > >
>>
>> > > > ______________________________________________
>>
>> > > > R-help using r-project.org mailing list -- To UNSUBSCRIBE and more, see
>>
>> > > > https://urldefense.com/v3/__https://stat.ethz.ch/mailman/listinfo/r-help__;!!Fou38LsQmgU!qcJ4z-vlMNzsa8XCsJUcuPOz8Vt12zsV_XaWpqXsyUYJBTlcNRonFPr7w7Ql3xqcDnZ9ZYC8JX72PW30DQ$
>>
>> > > > PLEASE do read the posting guide https://urldefense.com/v3/__http://www.R-project.org/posting-guide.html__;!!Fou38LsQmgU!qcJ4z-vlMNzsa8XCsJUcuPOz8Vt12zsV_XaWpqXsyUYJBTlcNRonFPr7w7Ql3xqcDnZ9ZYC8JX66rfmKvA$
>>
>> > > > and provide commented, minimal, self-contained, reproducible code.
>>
>> > >
>>
>> > > ______________________________________________
>>
>> > > R-help using r-project.org mailing list -- To UNSUBSCRIBE and more, see
>>
>> > > https://urldefense.com/v3/__https://stat.ethz.ch/mailman/listinfo/r-help__;!!Fou38LsQmgU!qcJ4z-vlMNzsa8XCsJUcuPOz8Vt12zsV_XaWpqXsyUYJBTlcNRonFPr7w7Ql3xqcDnZ9ZYC8JX72PW30DQ$
>>
>> > > PLEASE do read the posting guide https://urldefense.com/v3/__http://www.R-project.org/posting-guide.html__;!!Fou38LsQmgU!qcJ4z-vlMNzsa8XCsJUcuPOz8Vt12zsV_XaWpqXsyUYJBTlcNRonFPr7w7Ql3xqcDnZ9ZYC8JX66rfmKvA$
>>
>> > > and provide commented, minimal, self-contained, reproducible code.