[R] Error: C stack usage is too close to the limit when using list.files()

Sat Sep 28 19:51:25 CEST 2013

Thanks all -- ok, so the symbolic link issue is a distinct
possibility, but fundamentally doesn't solve the issue since most
users will have symbolic links on their machines SOMEPLACE, so a full
drive scan will run into these issues --  is list.files calling find,
or is it using a different algorithm?  This seems like a shortcoming
in the list.files algorithm -- is there a better solution (short of a
System call, which I'm still not sure will work on Macs without Xcode
-- a colleague of mine did NOT have Xcode, and reported not being able
to run find from the command line) -- perhaps a different package?

--j

On Fri, Sep 27, 2013 at 3:08 PM, William Dunlap <wdunlap at tibco.com> wrote:
> Toss a couple of extra files in there and you will see the output grow exponentially.
>
> % touch dir/IMPORTANT_1 dir/subdir/IMPORTANT_2
>
> and in R those two new files cause 82 more strings to appear in list.file's output:
>
>> nchar(list.files("dir", recursive=TRUE))
>  [1]  11  18  33  40  55  62  77  84  99 106 121 128 143 150 165 172 187 194 209
> [20] 216 231 238 253 260 275 282 297 304 319 326 341 348 363 370 385 392 407 414
> [39] 429 436 451 458 473 480 495 502 517 524 539 546 561 568 583 590 605 612 627
> [58] 634 649 656 671 678 693 700 715 722 737 744 759 766 781 788 803 810 825 832
> [77] 847 854 869 876 891 898 901
>
> 'find', by default, does not following symbolic links.
>
> % find dir
> dir
> dir/subdir
> dir/subdir/IMPORTANT_2
> dir/subdir/linkToUpperDir
> dir/IMPORTANT_1
>
> The -L option makes it follow them, but it won't follow loops:
>
> % find -L dir
> dir
> dir/subdir
> dir/subdir/IMPORTANT_2
> find: File system loop detected; `dir/subdir/linkToUpperDir' is part of the same file system loop as `dir'.
> dir/IMPORTANT_1
>
> Bill Dunlap
> Spotfire, TIBCO Software
> wdunlap tibco.com
>
>
>> -----Original Message-----
>> From: r-help-bounces at r-project.org [mailto:r-help-bounces at r-project.org] On Behalf
>> Of William Dunlap
>> Sent: Friday, September 27, 2013 12:56 PM
>> To: Jonathan Greenberg; r-help
>> Subject: Re: [R] Error: C stack usage is too close to the limit when using list.files()
>>
>> Do you have some symbolic links that make loops in your file system?
>> list.files() has problems with such loops and find does not.  E.g.,  on a Linux box:
>>
>> % cd /tmp
>> % mkdir dir dir/subdir
>> % cd dir/subdir
>> % ln -s ../../dir linkToUpperDir
>> % cd /tmp
>> % R --quiet
>> > list.files("dir", recursive=TRUE, full=TRUE)
>> [1]
>> "dir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkToU
>> pperDir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkT
>> oUpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/li
>> nkToUpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdi
>> r/linkToUpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir/su
>> bdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir
>> /subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkToUpper
>> Dir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkToUp
>> perDir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkTo
>> UpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/lin
>> kToUpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir"
>> > system("find dir")
>> dir
>> dir/subdir
>> dir/subdir/linkToUpperDir
>>
>> Bill Dunlap
>> Spotfire, TIBCO Software
>> wdunlap tibco.com
>>
>>
>> > -----Original Message-----
>> > From: r-help-bounces at r-project.org [mailto:r-help-bounces at r-project.org] On Behalf
>> > Of Jonathan Greenberg
>> > Sent: Friday, September 27, 2013 12:13 PM
>> > To: r-help
>> > Subject: [R] Error: C stack usage is too close to the limit when using list.files()
>> >
>> > R-helpers:
>> >
>> > I'm running a file search on my entire drive (Mac OS X) using:
>> >
>> > files_found <- list.files(dir="/",pattern=somepattern,recursive=TRUE,full.names=TRUE)
>> > where somepattern is a search pattern (which I have confirmed via a
>> > unix "find / -name somepattern" only returns ~ 3 results).
>> >
>> > I keep getting an error:
>> >
>> > Error: C stack usage is too close to the limit
>> >
>> > when running this command.  Any ideas on 1) how to fix this or 2) if
>> > there is an alternative to using list.files() to accomplish this
>> > search without resorting to an external package?
>> >
>> > Cheers!
>> >
>> > --jonathan
>> >
>> >
>> > --
>> > Jonathan A. Greenberg, PhD
>> > Assistant Professor
>> > Global Environmental Analysis and Remote Sensing (GEARS) Laboratory
>> > Department of Geography and Geographic Information Science
>> > University of Illinois at Urbana-Champaign
>> > 259 Computing Applications Building, MC-150
>> > 605 East Springfield Avenue
>> > Champaign, IL  61820-6371
>> > Phone: 217-300-1924
>> > http://www.geog.illinois.edu/~jgrn/
>> > AIM: jgrn307, MSN: jgrn307 at hotmail.com, Gchat: jgrn307, Skype: jgrn3007
>> >
>> > ______________________________________________
>> > R-help at r-project.org mailing list
>> > https://stat.ethz.ch/mailman/listinfo/r-help
>> > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
>> > and provide commented, minimal, self-contained, reproducible code.
>>
>> ______________________________________________
>> R-help at r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-help
>> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
>> and provide commented, minimal, self-contained, reproducible code.


-- 
Jonathan A. Greenberg, PhD
Assistant Professor
Global Environmental Analysis and Remote Sensing (GEARS) Laboratory
Department of Geography and Geographic Information Science
University of Illinois at Urbana-Champaign
259 Computing Applications Building, MC-150
605 East Springfield Avenue
Champaign, IL  61820-6371
Phone: 217-300-1924
http://www.geog.illinois.edu/~jgrn/
AIM: jgrn307, MSN: jgrn307 at hotmail.com, Gchat: jgrn307, Skype: jgrn3007