[R-sig-Geo] still struggling with FTP download on MODIS

[Tomislav Hengl]

> -----Original Message-----
> From: r-sig-geo-bounces at stat.math.ethz.ch [mailto:r-sig-geo-bounces at stat.math.ethz.ch] On Behalf
> Of Alexandre VILLERS
> Sent: Wednesday, July 15, 2009 10:03 AM
> To: Aide R SIG GEO
> Subject: [R-sig-Geo] still struggling with FTP download on MODIS
> 
> Dear all,
> 
> I'm still not sure that this list is the right place for that. So if
> those questions bother you, please give me the name of the list I should
> contact.
> 
> Here a version of the script I'm using to download MODIS data from NSIDC
> (weekly snow cover)
> 
> Tsnow <-
> "ftp://anonymous:test@n4ftl01u.ecs.nasa.gov/SAN/MOST/MOD10A2.005/"
> ###server of NSIDC with directories / id and password
> myfolder<-paste(tempdir(),"\\" , sep="") #the folder to download data
> 
> #####-getting the list of the folders in Tsnow (thanks to Barry Rowlingson)
> resp <-getURL(Tsnow)
>   items <- strsplit(resp,"\r\n")[[1]]
>   folderLines <- items[substr(items,1,1)=='d']
>   lastBit <- function(x){x[length(x)]}
>   dates <- unlist(lapply(strsplit(folderLines," "),lastBit))
>   datesK<-data.frame(dates=dates, keep=rep(NA,length(dates)))
> 
> ##### select months for which I want the data
> months<-c("01","02","03","04","05","06","10", "11", "12") #months for
> which I want the data
>         for (i in months){ #loop to keep only the dates I'm interested in
>               garde<-grep(datesK$dates, pattern=paste(".",i,".", sep=""))
>               datesK$keep[garde]<-1
>               }
> datesK<-subset(datesK, datesK$keep==1) # keep the dates of interest
> 
> for (i in 1:length(datesK[,1])){    ## loop to download the tiles of
> interest
>         getlist1 <- strsplit(getURL(paste(Tsnow, datesK[i,][[1]], "/",
> sep=""),.opts=curlOptions(ftplistonly=TRUE)), "\r\n")[[1]] #get the
> files in the folder i
>         BLOCKtemp <- getlist1[grep(getlist1,
> pattern="MOD10A2.********.h18v04.005.*************.hdf")[1]] #select the
> tile of interest (in my case h18v04.005)
>         download.file(paste(Tsnow, datesK[i,][[1]], "/",
> BLOCKtemp,sep=""), destfile=paste(myfolder, BLOCKtemp, sep=""),
> mode='wb', method='wget') #download the tile
>         }
> 
> So here is my problem. I can't get into some specifc folders and receive
> the error message.
> Error in curlPerform(curl = curl, .opts = opts, .encoding = .encoding) :
>   server did not report OK, got 550

The ftp error code "550" means: "Requested action not taken. File unavailable (e.g., file not found,
no access).", which means that the ftp server is not allowing you access.

Did you try checking your wget tool? I remember I had headaches before I manage to get the wget
running without problems. Make sure you set:

> options(download.file.method="auto")

wget.exe you need to put in your Windows directory folder (Note: make sure you disable your
antivirus tools such as Norton or McAfee otherwise it might block wget from running);


> For example, i=4 (folder "2000.03.13"). And this is always true,
> suggesting it's not a problem of connection. The error appears as soon
> as getURL is called in the loop.
> I succeed in downloading for some values of i (manually) but when
> looping, it stops on those "black" folders. I've checked directly from
> filezilla or my web browser: I can get into those folder and download
> the tiles.
> 
> The tiles are pretty small (150 to 450 Ko). So if someone could try from
> his place to see if he gets the same error message for the same value of
> i (4, 28,31, etc.).
> Since I have 550 tiles to download and process, I would prefer not to do
> that manually (and I can't get to use the tool proposed by the NSIDC for
> the moment).
> 
> Thanks for the help
> 
> Alex
> 
> P.S.: my next question will deal with some projection problem !!!
> 
> --
> Alexandre Villers
> PhD Student
> Team "Biodiversity"
> Centre d'Etudes Biologiques de Chizé-CNRS UPR1934
> 79360 Beauvoir sur Niort
> 
> Phone +33 (0)5 49 09 96 13
> Fax   +33 (0)5 49 09 65 26
> 
> 
> 
> 
> __________ Information from ESET Mail Security, version of virus signature database 4244
> (20090715) __________
> 
> The message was checked by ESET Mail Security.
> http://www.eset.com
> 
> _______________________________________________
> R-sig-Geo mailing list
> R-sig-Geo at stat.math.ethz.ch
> https://stat.ethz.ch/mailman/listinfo/r-sig-geo

T. Hengl
http://spatial-analyst.net