[R] multi day R run crashed - is there a log

[Sarah Goslee]

Hi,

On Wed, Aug 25, 2010 at 8:11 AM, Martin Tomko <martin.tomko at geo.uzh.ch> wrote:
> Hi Sarah,
> thank you very much for your answer. I have been spitting things out on
> screen, but unfortunately I have not run it as a batch log, but in the
> interactive window, so when the GUI crashed I was left without trace... I
> guess I should google how to run it from a batch.

I have  no idea how to do that in Windows, but I'm sure it's possible. :)
That way the things written to the screen will be saved in a file instead.

> I should also explore using RData, I have ony been using csv files and
> tables so far, it seems that can also bring some added performance.

If you need to save *everything*, RData is what you get when you use save(),
or when you close a session and choose y to saving the data. It can be
read in using load(). That's one way to be able to pick up where you left
off.

> As the main output of my process is a matrix, I would really need to append
> to a matrix after each iteration. I have identified the write.table append
> parameter-based solution, but that would only append rows. Is there a way to
> slowly "grow" a matrix in both directions, meaning append columns as well
> (it is a big distance matrix).

AFAIK, you can't append columns that way because of the way text files are
written to disk. You'd need to rewrite the whole thing, or possibly write it out
in lower triangular format with NA values as padding (assuming it's a symmetric
distance).

Or for that matter, you could just write it out as a really long
vector, and turn
it back into a matrix later if you need to read the saved file in after a crash.

I'd recommend saving whatever variables are needed so that you can pick up
exactly where you left off, if possible. Much nicer to pick up 12 hours in than
to start over from the beginning.

Not R, but I just finished a 5-week batch job. You can bet that I put a lot of
thought into incremental save points and how to resume after an unexpected
halt!

Sarah

-- 
Sarah Goslee
http://www.functionaldiversity.org