[R] Error: C stack usage is too close to the limit when using list.files()

[William Dunlap]

The issue is not symbolic links per se, but ones that form loops.
Note that you can detect such loops by running 'find -L ...' and 
looking for the error messages.  (find by default does not follow
any symbolic links, which can be a problem also.)

It is a shortcoming of the current version of list.files().

Bill Dunlap
Spotfire, TIBCO Software
wdunlap tibco.com


> -----Original Message-----
> From: jgrn307 at gmail.com [mailto:jgrn307 at gmail.com] On Behalf Of Jonathan
> Greenberg
> Sent: Saturday, September 28, 2013 10:51 AM
> To: William Dunlap
> Cc: r-help
> Subject: Re: [R] Error: C stack usage is too close to the limit when using list.files()
> 
> Thanks all -- ok, so the symbolic link issue is a distinct
> possibility, but fundamentally doesn't solve the issue since most
> users will have symbolic links on their machines SOMEPLACE, so a full
> drive scan will run into these issues --  is list.files calling find,
> or is it using a different algorithm?  This seems like a shortcoming
> in the list.files algorithm -- is there a better solution (short of a
> System call, which I'm still not sure will work on Macs without Xcode
> -- a colleague of mine did NOT have Xcode, and reported not being able
> to run find from the command line) -- perhaps a different package?
> 
> --j
> 
> On Fri, Sep 27, 2013 at 3:08 PM, William Dunlap <wdunlap at tibco.com> wrote:
> > Toss a couple of extra files in there and you will see the output grow exponentially.
> >
> > % touch dir/IMPORTANT_1 dir/subdir/IMPORTANT_2
> >
> > and in R those two new files cause 82 more strings to appear in list.file's output:
> >
> >> nchar(list.files("dir", recursive=TRUE))
> >  [1]  11  18  33  40  55  62  77  84  99 106 121 128 143 150 165 172 187 194 209
> > [20] 216 231 238 253 260 275 282 297 304 319 326 341 348 363 370 385 392 407 414
> > [39] 429 436 451 458 473 480 495 502 517 524 539 546 561 568 583 590 605 612 627
> > [58] 634 649 656 671 678 693 700 715 722 737 744 759 766 781 788 803 810 825 832
> > [77] 847 854 869 876 891 898 901
> >
> > 'find', by default, does not following symbolic links.
> >
> > % find dir
> > dir
> > dir/subdir
> > dir/subdir/IMPORTANT_2
> > dir/subdir/linkToUpperDir
> > dir/IMPORTANT_1
> >
> > The -L option makes it follow them, but it won't follow loops:
> >
> > % find -L dir
> > dir
> > dir/subdir
> > dir/subdir/IMPORTANT_2
> > find: File system loop detected; `dir/subdir/linkToUpperDir' is part of the same file
> system loop as `dir'.
> > dir/IMPORTANT_1
> >
> > Bill Dunlap
> > Spotfire, TIBCO Software
> > wdunlap tibco.com
> >
> >
> >> -----Original Message-----
> >> From: r-help-bounces at r-project.org [mailto:r-help-bounces at r-project.org] On
> Behalf
> >> Of William Dunlap
> >> Sent: Friday, September 27, 2013 12:56 PM
> >> To: Jonathan Greenberg; r-help
> >> Subject: Re: [R] Error: C stack usage is too close to the limit when using list.files()
> >>
> >> Do you have some symbolic links that make loops in your file system?
> >> list.files() has problems with such loops and find does not.  E.g.,  on a Linux box:
> >>
> >> % cd /tmp
> >> % mkdir dir dir/subdir
> >> % cd dir/subdir
> >> % ln -s ../../dir linkToUpperDir
> >> % cd /tmp
> >> % R --quiet
> >> > list.files("dir", recursive=TRUE, full=TRUE)
> >> [1]
> >>
> "dir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkToU
> >>
> pperDir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkT
> >>
> oUpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/li
> >>
> nkToUpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdi
> >>
> r/linkToUpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir/su
> >>
> bdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir
> >>
> /subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkToUpper
> >>
> Dir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkToUp
> >>
> perDir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkTo
> >>
> UpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/lin
> >> kToUpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir"
> >> > system("find dir")
> >> dir
> >> dir/subdir
> >> dir/subdir/linkToUpperDir
> >>
> >> Bill Dunlap
> >> Spotfire, TIBCO Software
> >> wdunlap tibco.com
> >>
> >>
> >> > -----Original Message-----
> >> > From: r-help-bounces at r-project.org [mailto:r-help-bounces at r-project.org] On
> Behalf
> >> > Of Jonathan Greenberg
> >> > Sent: Friday, September 27, 2013 12:13 PM
> >> > To: r-help
> >> > Subject: [R] Error: C stack usage is too close to the limit when using list.files()
> >> >
> >> > R-helpers:
> >> >
> >> > I'm running a file search on my entire drive (Mac OS X) using:
> >> >
> >> > files_found <-
> list.files(dir="/",pattern=somepattern,recursive=TRUE,full.names=TRUE)
> >> > where somepattern is a search pattern (which I have confirmed via a
> >> > unix "find / -name somepattern" only returns ~ 3 results).
> >> >
> >> > I keep getting an error:
> >> >
> >> > Error: C stack usage is too close to the limit
> >> >
> >> > when running this command.  Any ideas on 1) how to fix this or 2) if
> >> > there is an alternative to using list.files() to accomplish this
> >> > search without resorting to an external package?
> >> >
> >> > Cheers!
> >> >
> >> > --jonathan
> >> >
> >> >
> >> > --
> >> > Jonathan A. Greenberg, PhD
> >> > Assistant Professor
> >> > Global Environmental Analysis and Remote Sensing (GEARS) Laboratory
> >> > Department of Geography and Geographic Information Science
> >> > University of Illinois at Urbana-Champaign
> >> > 259 Computing Applications Building, MC-150
> >> > 605 East Springfield Avenue
> >> > Champaign, IL  61820-6371
> >> > Phone: 217-300-1924
> >> > http://www.geog.illinois.edu/~jgrn/
> >> > AIM: jgrn307, MSN: jgrn307 at hotmail.com, Gchat: jgrn307, Skype: jgrn3007
> >> >
> >> > ______________________________________________
> >> > R-help at r-project.org mailing list
> >> > https://stat.ethz.ch/mailman/listinfo/r-help
> >> > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> >> > and provide commented, minimal, self-contained, reproducible code.
> >>
> >> ______________________________________________
> >> R-help at r-project.org mailing list
> >> https://stat.ethz.ch/mailman/listinfo/r-help
> >> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> >> and provide commented, minimal, self-contained, reproducible code.
> 
> 
> 
> --
> Jonathan A. Greenberg, PhD
> Assistant Professor
> Global Environmental Analysis and Remote Sensing (GEARS) Laboratory
> Department of Geography and Geographic Information Science
> University of Illinois at Urbana-Champaign
> 259 Computing Applications Building, MC-150
> 605 East Springfield Avenue
> Champaign, IL  61820-6371
> Phone: 217-300-1924
> http://www.geog.illinois.edu/~jgrn/
> AIM: jgrn307, MSN: jgrn307 at hotmail.com, Gchat: jgrn307, Skype: jgrn3007