[R] multi day R run crashed - is there a log

[Martin Tomko]

Hi ,
thanks, the lower.tri idea is I guess the best way. Will try that.
Cheers
Martin

On 8/25/2010 2:21 PM, Sarah Goslee wrote:
> Hi,
>
> On Wed, Aug 25, 2010 at 8:11 AM, Martin Tomko<martin.tomko at geo.uzh.ch>  wrote:
>    
>> Hi Sarah,
>> thank you very much for your answer. I have been spitting things out on
>> screen, but unfortunately I have not run it as a batch log, but in the
>> interactive window, so when the GUI crashed I was left without trace... I
>> guess I should google how to run it from a batch.
>>      
> I have  no idea how to do that in Windows, but I'm sure it's possible. :)
> That way the things written to the screen will be saved in a file instead.
>
>    
>> I should also explore using RData, I have ony been using csv files and
>> tables so far, it seems that can also bring some added performance.
>>      
> If you need to save *everything*, RData is what you get when you use save(),
> or when you close a session and choose y to saving the data. It can be
> read in using load(). That's one way to be able to pick up where you left
> off.
>
>    
>> As the main output of my process is a matrix, I would really need to append
>> to a matrix after each iteration. I have identified the write.table append
>> parameter-based solution, but that would only append rows. Is there a way to
>> slowly "grow" a matrix in both directions, meaning append columns as well
>> (it is a big distance matrix).
>>      
> AFAIK, you can't append columns that way because of the way text files are
> written to disk. You'd need to rewrite the whole thing, or possibly write it out
> in lower triangular format with NA values as padding (assuming it's a symmetric
> distance).
>
> Or for that matter, you could just write it out as a really long
> vector, and turn
> it back into a matrix later if you need to read the saved file in after a crash.
>
> I'd recommend saving whatever variables are needed so that you can pick up
> exactly where you left off, if possible. Much nicer to pick up 12 hours in than
> to start over from the beginning.
>
> Not R, but I just finished a 5-week batch job. You can bet that I put a lot of
> thought into incremental save points and how to resume after an unexpected
> halt!
>
> Sarah
>
>    


-- 
Martin Tomko
Postdoctoral Research Assistant

Geographic Information Systems Division
Department of Geography
University of Zurich - Irchel
Winterthurerstr. 190
CH-8057 Zurich, Switzerland

email: 	martin.tomko at geo.uzh.ch
site:	http://www.geo.uzh.ch/~mtomko
mob: 	+41-788 629 558
tel: 	+41-44-6355256
fax: 	+41-44-6356848