[R] iterators : checkFunc with ireadLines

[William Michels]

Hi Laurent,

Seeking to give you an "R-only" solution, I thought the read.fwf()
function might be useful (to read-in your first column of data, only).
However Jeff is correct that this is a poor strategy, since read.fwf()
reads the entire file into R (documented in "Fixed-width-format
files", Section 2.2: R Data Import/Export Manual).

Jeff has suggested a number of packages, as well as using a database.
Ivan Krylov has posted answers using grep, awk and perl (perl5--to
disambiguate). [In point of fact, the R Data Import/Export Manual
suggests using perl]. Similar to Ivan, I've posted code below using
the Raku programming language (the language formerly known as Perl6).
Regexes are claimed to be more readable, but are currently very slow
in Raku. However on the plus side, the language is designed to handle
Unicode gracefully:

> # pipe() using raku-grep on Laurent's data (sep=mult whitespace):
> con_obj1 <- pipe(paste("raku -e '.put for lines.grep( / ^^N053 | ^^N163 /, :p );' ", "Laurents.txt"), open="rt");
> p6_import_a <- scan(file=con_obj1, what=list("","","","","","","","","",""), flush=TRUE, multi.line=FALSE, quiet=TRUE);
> close(con_obj1);
> as.data.frame(sapply(p6_import_a, t), stringsAsFactors=FALSE);
  V1   V2        V3        V4        V5        V6        V7        V8
      V9       V10
1  2 N053 -0.014083 -0.004741  0.001443 -0.010152 -0.012996 -0.005337
-0.008738 -0.015094
2  4 N163 -0.054023 -0.049345 -0.037158  -0.04112 -0.044612 -0.036953
-0.036061 -0.044516
>
> # pipe() using raku-grep "starts-with" to find genbankID ( >3GB TSV file)
> # "lines[0..5]" restricts raku to reading first 6 lines!
> # change "lines[0..5]" to "lines" to run raku code on whole file:
> con_obj2 <- pipe(paste("raku -e '.put for lines[0..5].grep( *.starts-with(q[A00145]), :p);' ", "genbankIDs_3GB.tsv"), "rt");
> p6_import_b <- read.table(con_obj2, sep="\t");
> close(con_obj2)
> p6_import_b
  V1     V2       V3          V4 V5
1  4 A00145 A00145.1 IFN-alpha A NA
>
> # unicode test using R's system() function:
> try(system("raku -ne '.grep( /  你好  |  こんにちは  |  مرحبا  |  Привет  /, :v ).put;'  hello_7lang.txt", intern = TRUE, ignore.stderr = FALSE))
[1] ""                    ""                    ""
"你好 Chinese"
[5] "こんにちは Japanese" "مرحبا Arabic"        "Привет Russian"
>

[special thanks to Brad Gilbert, Joseph Brenner and others on the
perl6-users mailing list. All errors above are my own.]

HTH, Bill.

W. Michels, Ph.D.




On Fri, May 22, 2020 at 4:48 AM Laurent Rhelp <LaurentRHelp using free.fr> wrote:
>
> Hi Ivan,
>    Endeed, it is a good idea. I am under MSwindows but I can use the
> bash command I use with git. I will see how to do that with the unix
> command lines.
>
>
> Le 20/05/2020 à 09:46, Ivan Krylov a écrit :
> > Hi Laurent,
> >
> > I am not saying this will work every time and I do recognise that this
> > is very different from a more general solution that you had envisioned,
> > but if you are on an UNIX-like system or have the relevant utilities
> > installed and on the %PATH% on Windows, you can filter the input file
> > line-by-line using a pipe and an external program:
> >
> > On Sun, 17 May 2020 15:52:30 +0200
> > Laurent Rhelp <LaurentRHelp using free.fr> wrote:
> >
> >> # sensors to keep
> >> sensors <-  c("N053", "N163")
> > # filter on the beginning of the line
> > i <- pipe("grep -E '^(N053|N163)' test.txt")
> > # or:
> > # filter on the beginning of the given column
> > # (use $2 for the second column, etc.)
> > i <- pipe("awk '($1 ~ \"^(N053|N163)\")' test.txt")
> > # or:
> > # since your message is full of Unicode non-breaking spaces, I have to
> > # bring in heavier machinery to handle those correctly;
> > # only this solution manages to match full column values
> > # (here you can also use $F[1] for second column and so on)
> > i <- pipe("perl -CSD -F'\\s+' -lE \\
> >   'print join qq{\\t}, @F if $F[0] =~ /^(N053|N163)$/' \\
> >   test.txt
> > ")
> > lines <- read.table(i) # closes i when done
> >
> > The downside of this approach is having to shell-escape the command
> > lines, which can become complicated, and choosing between use of regular
> > expressions and more wordy programs (Unicode whitespace in the input
> > doesn't help, either).
> >
>
>
> --
> L'absence de virus dans ce courrier électronique a été vérifiée par le logiciel antivirus Avast.
> https://www.avast.com/antivirus
>
> ______________________________________________
> R-help using r-project.org mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.