[R] Error: C stack usage is too close to the limit when using list.files()

Thu Dec 5 06:20:07 CET 2013

FYI, in R.utils (>= 1.28.4) you can use listDirectory() to control how
deep the recursion goes, which would give you protection against your
problem, e.g.

  R.utils::listDirectory("dir", recursive=5L)

where recursive=0L is equivalent to recursive=FALSE.  Using
recursive=TRUE corresponds to recursive=+Inf, that is infinite depth.
listDirectory() accepts similar arguments that list.files() does.

/Henrik

On Sat, Sep 28, 2013 at 5:06 PM, William Dunlap <wdunlap at tibco.com> wrote:
> The issue is not symbolic links per se, but ones that form loops.
> Note that you can detect such loops by running 'find -L ...' and
> looking for the error messages.  (find by default does not follow
> any symbolic links, which can be a problem also.)
>
> It is a shortcoming of the current version of list.files().
>
> Bill Dunlap
> Spotfire, TIBCO Software
> wdunlap tibco.com
>
>
>> -----Original Message-----
>> From: jgrn307 at gmail.com [mailto:jgrn307 at gmail.com] On Behalf Of Jonathan
>> Greenberg
>> Sent: Saturday, September 28, 2013 10:51 AM
>> To: William Dunlap
>> Cc: r-help
>> Subject: Re: [R] Error: C stack usage is too close to the limit when using list.files()
>>
>> Thanks all -- ok, so the symbolic link issue is a distinct
>> possibility, but fundamentally doesn't solve the issue since most
>> users will have symbolic links on their machines SOMEPLACE, so a full
>> drive scan will run into these issues --  is list.files calling find,
>> or is it using a different algorithm?  This seems like a shortcoming
>> in the list.files algorithm -- is there a better solution (short of a
>> System call, which I'm still not sure will work on Macs without Xcode
>> -- a colleague of mine did NOT have Xcode, and reported not being able
>> to run find from the command line) -- perhaps a different package?
>>
>> --j
>>
>> On Fri, Sep 27, 2013 at 3:08 PM, William Dunlap <wdunlap at tibco.com> wrote:
>> > Toss a couple of extra files in there and you will see the output grow exponentially.
>> >
>> > % touch dir/IMPORTANT_1 dir/subdir/IMPORTANT_2
>> >
>> > and in R those two new files cause 82 more strings to appear in list.file's output:
>> >
>> >> nchar(list.files("dir", recursive=TRUE))
>> >  [1]  11  18  33  40  55  62  77  84  99 106 121 128 143 150 165 172 187 194 209
>> > [20] 216 231 238 253 260 275 282 297 304 319 326 341 348 363 370 385 392 407 414
>> > [39] 429 436 451 458 473 480 495 502 517 524 539 546 561 568 583 590 605 612 627
>> > [58] 634 649 656 671 678 693 700 715 722 737 744 759 766 781 788 803 810 825 832
>> > [77] 847 854 869 876 891 898 901
>> >
>> > 'find', by default, does not following symbolic links.
>> >
>> > % find dir
>> > dir
>> > dir/subdir
>> > dir/subdir/IMPORTANT_2
>> > dir/subdir/linkToUpperDir
>> > dir/IMPORTANT_1
>> >
>> > The -L option makes it follow them, but it won't follow loops:
>> >
>> > % find -L dir
>> > dir
>> > dir/subdir
>> > dir/subdir/IMPORTANT_2
>> > find: File system loop detected; `dir/subdir/linkToUpperDir' is part of the same file
>> system loop as `dir'.
>> > dir/IMPORTANT_1
>> >
>> > Bill Dunlap
>> > Spotfire, TIBCO Software
>> > wdunlap tibco.com
>> >
>> >
>> >> -----Original Message-----
>> >> From: r-help-bounces at r-project.org [mailto:r-help-bounces at r-project.org] On
>> Behalf
>> >> Of William Dunlap
>> >> Sent: Friday, September 27, 2013 12:56 PM
>> >> To: Jonathan Greenberg; r-help
>> >> Subject: Re: [R] Error: C stack usage is too close to the limit when using list.files()
>> >>
>> >> Do you have some symbolic links that make loops in your file system?
>> >> list.files() has problems with such loops and find does not.  E.g.,  on a Linux box:
>> >>
>> >> % cd /tmp
>> >> % mkdir dir dir/subdir
>> >> % cd dir/subdir
>> >> % ln -s ../../dir linkToUpperDir
>> >> % cd /tmp
>> >> % R --quiet
>> >> > list.files("dir", recursive=TRUE, full=TRUE)
>> >> [1]
>> >>
>> "dir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkToU
>> >>
>> pperDir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkT
>> >>
>> oUpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/li
>> >>
>> nkToUpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdi
>> >>
>> r/linkToUpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir/su
>> >>
>> bdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir
>> >>
>> /subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkToUpper
>> >>
>> Dir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkToUp
>> >>
>> perDir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkTo
>> >>
>> UpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/lin
>> >> kToUpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir"
>> >> > system("find dir")
>> >> dir
>> >> dir/subdir
>> >> dir/subdir/linkToUpperDir
>> >>
>> >> Bill Dunlap
>> >> Spotfire, TIBCO Software
>> >> wdunlap tibco.com
>> >>
>> >>
>> >> > -----Original Message-----
>> >> > From: r-help-bounces at r-project.org [mailto:r-help-bounces at r-project.org] On
>> Behalf
>> >> > Of Jonathan Greenberg
>> >> > Sent: Friday, September 27, 2013 12:13 PM
>> >> > To: r-help
>> >> > Subject: [R] Error: C stack usage is too close to the limit when using list.files()
>> >> >
>> >> > R-helpers:
>> >> >
>> >> > I'm running a file search on my entire drive (Mac OS X) using:
>> >> >
>> >> > files_found <-
>> list.files(dir="/",pattern=somepattern,recursive=TRUE,full.names=TRUE)
>> >> > where somepattern is a search pattern (which I have confirmed via a
>> >> > unix "find / -name somepattern" only returns ~ 3 results).
>> >> >
>> >> > I keep getting an error:
>> >> >
>> >> > Error: C stack usage is too close to the limit
>> >> >
>> >> > when running this command.  Any ideas on 1) how to fix this or 2) if
>> >> > there is an alternative to using list.files() to accomplish this
>> >> > search without resorting to an external package?
>> >> >
>> >> > Cheers!
>> >> >
>> >> > --jonathan
>> >> >
>> >> >
>> >> > --
>> >> > Jonathan A. Greenberg, PhD
>> >> > Assistant Professor
>> >> > Global Environmental Analysis and Remote Sensing (GEARS) Laboratory
>> >> > Department of Geography and Geographic Information Science
>> >> > University of Illinois at Urbana-Champaign
>> >> > 259 Computing Applications Building, MC-150
>> >> > 605 East Springfield Avenue
>> >> > Champaign, IL  61820-6371
>> >> > Phone: 217-300-1924
>> >> > http://www.geog.illinois.edu/~jgrn/
>> >> > AIM: jgrn307, MSN: jgrn307 at hotmail.com, Gchat: jgrn307, Skype: jgrn3007
>> >> >
>> >> > ______________________________________________
>> >> > R-help at r-project.org mailing list
>> >> > https://stat.ethz.ch/mailman/listinfo/r-help
>> >> > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
>> >> > and provide commented, minimal, self-contained, reproducible code.
>> >>
>> >> ______________________________________________
>> >> R-help at r-project.org mailing list
>> >> https://stat.ethz.ch/mailman/listinfo/r-help
>> >> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
>> >> and provide commented, minimal, self-contained, reproducible code.
>>
>>
>>
>> --
>> Jonathan A. Greenberg, PhD
>> Assistant Professor
>> Global Environmental Analysis and Remote Sensing (GEARS) Laboratory
>> Department of Geography and Geographic Information Science
>> University of Illinois at Urbana-Champaign
>> 259 Computing Applications Building, MC-150
>> 605 East Springfield Avenue
>> Champaign, IL  61820-6371
>> Phone: 217-300-1924
>> http://www.geog.illinois.edu/~jgrn/
>> AIM: jgrn307, MSN: jgrn307 at hotmail.com, Gchat: jgrn307, Skype: jgrn3007
>
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.