Bill, Jim and Martin,

Great! The code is much faster and even looks more R-ish. I'm very new to R
and have some difficulties getting rid my procedural programming habits.

Many thanks to all for the great help.

Olivier.

On Wed, Mar 18, 2009 at 9:38 PM, William Dunlap <wdunlap@tibco.com> wrote:

> Olivier,
> You can profile R code with Rprof().  E.g.,
>  Rprof(tmp<-tempfile()) # start profiling, saving results in a file
>  ... run your code: sapply(x, pn_formatting) ...
>  Rprof()   # stop profiling
>  summaryRprof(tmp) # analyze the file and present results in a pair of
> data.frames
>  $by.self
>                       self.time self.pct total.time total.pct
>  sub                       2.26     24.3       2.26      24.3
>  structure                 0.68      7.3       1.42      15.3
>  FUN                       0.66      7.1       8.82      94.8
>  withCallingHandlers       0.62      6.7       4.14      44.5
>  paste                     0.54      5.8       0.56       6.0
>  toupper                   0.44      4.7       0.46       4.9
>  as.integer                0.42      4.5       3.34      35.9
>  makeRestartList           0.38      4.1       1.36      14.6
>  unlist                    0.34      3.7       0.40       4.3
>  substr                    0.30      3.2       0.36       3.9
>  ...
>  $by.total
>                       total.time total.pct self.time self.pct
>  sapply                     9.30     100.0      0.00      0.0
>  lapply                     8.88      95.5      0.08      0.9
>  FUN                        8.82      94.8      0.66      7.1
>  digits_only                4.18      44.9      0.02      0.2
>  suppressWarnings           4.16      44.7      0.02      0.2
>  withCallingHandlers        4.14      44.5      0.62      6.7
>  as.integer                 3.34      35.9      0.42      4.5
>  withRestarts               2.92      31.4      0.10      1.1
>  .signalSimpleWarning       2.92      31.4      0.00      0.0
>  trim                       2.34      25.2      0.08      0.9
>  sub                        2.26      24.3      2.26     24.3
>
> You didn't mention that you used sapply() on your pn_formatting
> function, which I think you must have since you use a non-vectorized
> if statement in it.  If you vectorize the numeric/non-numeric choice,
> as in the following code, you get a huge speedup because you don't
> have to use sapply:
>
> pn_formatting1 <-
> function(pn_in) {
>  pn_out = trim(pn_in)
>   numeric <- digits_only(pn_out)
>  pn_out[!numeric] <- toupper(pn_out[!numeric])
>  pn_out[numeric] <- {
>    # Zero padding
>    tmp <- paste("000000000000000000", pn_out[numeric], sep="")
>    pn_len <- nchar(tmp)
>    substr(tmp, pn_len - 17, pn_len)
>  }
>  pn_out
> }
>
> Jim's code is a bit cleaner (to my taste) but runs at the same
> speed as yours after this simple modification.  His contains an
> error, in that it uses integer subscripts and does not check that
> there are at least one numeric entry in the input (in that case
> pn_in[-which.num] returns all of pn_in and sprintf() dies because
> one of its arguments is 0-long).
>
> Bill Dunlap
> TIBCO Software Inc - Spotfire Division
> wdunlap tibco.com
> ------------------------------------------------------------------------
> -----------
> [R] Profiling question: string formatting extremely slow
>
> jim holtman jholtman at gmail.com
> Wed Mar 18 18:09:37 CET 2009
> Previous message: [R] Profiling question: string formatting extremely
> slow
> Next message: [R] Updated R on Debian testing machine...
> Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
> Try this way.  Took less than 1 second for 50,000
>
> > system.time({
> +     x <- sample(50000)  # test data
> +     x[sample(50000,10000)] <- 'asdfasdf'  # characters strings
> +     which.num <- grep("^[ 0-9]+$", x)  # find numbers
> +     # convert to leading 0
> +     x[which.num] <- sprintf("%018.0f", as.numeric(x[which.num]))
> +     x[-which.num] <- toupper(x[-which.num])
> + })
>   user  system elapsed
>   0.25    0.00    0.25
> >
> >
> >
> > head(x,30)
>  [1] "000000000000026550" "000000000000019100" "000000000000045961"
> "000000000000031473" "000000000000005031" "000000000000012266"
>  [7] "000000000000034418" "000000000000042279" "000000000000041193"
> "ASDFASDF"           "000000000000005760" "000000000000035659"
> [13] "ASDFASDF"           "000000000000008420" "000000000000042220"
> "ASDFASDF"           "000000000000039903" "000000000000032234"
> [19] "000000000000024125" "000000000000032970" "000000000000006814"
> "000000000000000215" "ASDFASDF"           "000000000000045239"
> [25] "ASDFASDF"           "ASDFASDF"           "000000000000043065"
> "ASDFASDF"           "000000000000007642" "000000000000019196"
> >
>
>
> On Wed, Mar 18, 2009 at 12:16 PM, Olivier Boudry
> <olivier.boudry at gmail.com> wrote:
> > Hi all,
> >
> > I'm using R to find duplicates in a set of 6 files containing Part
> Number
> > information. Before applying the intersect method to identify the
> duplicates
> > I need to normalize the P/Ns. Converting the P/N to uppercase if
> > alphanumerical and applying an 18 char long zero padding if numerical.
> >
> > When I apply the pn_formatting function (see code below) to "Part
> Number"
> > column of the data.frame (character vectors up to 18 char long) it
> consumes
> > a lot of memory, my computer (Windows XP SP3) starts to swap memory,
> CPU
> > goes to zero and completion takes hours to complete. Part Number
> columns can
> > have from 7'000 to 80'000 records and I've never got enough patience
> to wait
> > for completion of more than 17'000 records.
> >
> > Is there a way to find out which of the function used below is the
> > bottleneck, as.integer, is.na, sub, paste, nchar, toupper? Is there a
> > profiler for R and if yes where could I find some documentation on how
> to
> > use it?
> >
> > The code:
> >
> > # String contains digits only (can be converted to an integer)
> > digits_only <- function(x) { suppressWarnings(!is.na(as.integer(x))) }
> >
> > # Remove blanks at both ends of a string
> > trim <- function (x) {
> >  sub("^\\s+((.*\\S)\\s+)?$", "\\2", x)
> > }
> >
> > # P/N formatting
> > pn_formatting <- function(pn_in) {
> >
> >  pn_out = trim(pn_in)
> >  if (digits_only(pn_out)) {
> >
> >    # Zero padding
> >    pn_out <- paste("000000000000000000", pn_out, sep="")
> >    pn_len <- nchar(pn_out)
> >    pn_out <- substr(pn_out, pn_len - 17, pn_len)
> >
> >  } else {
> >    # Uppercase
> >    pn_out <- toupper(pn_out)
> >  }
> >  pn_out
> > }
> >
> > Thanks,
> >
> > Olivier.
> >
> >        [[alternative HTML version deleted]]
> >
> > ______________________________________________
> > R-help at r-project.org mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-help
> > PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> > and provide commented, minimal, self-contained, reproducible code.
> >
>
>
>
> --
> Jim Holtman
> Cincinnati, OH
> +1 513 646 9390
>
> What is the problem that you are trying to solve?
>

	[[alternative HTML version deleted]]


