[R-sig-hpc] Error: serialization is too large to store in a raw vector

[Stephen Weston]

I don't see why making U133_ps a factor would cause the serialization
error that you reported.  I wasn't able to reproduce it.  I suspect that
something else was going on as well.

- Steve


On Wed, Aug 11, 2010 at 12:38 AM, Fong Chun Chan
<fongchun at interchange.ubc.ca> wrote:
> Thanks for the reply Steve.  I actually finally managed to figure out what
> was going on.  The U133_ps was a single column factor actually.  Once I
> converted it into a single column character vector, I didn't have any more
> problems
>
> Do you know why the factor class was causing so many issues?
>
> Fong
>
>
> On Tue, Aug 10, 2010 at 7:03 PM, Stephen Weston <stephen.b.weston at gmail.com>
> wrote:
>>
>> Hi Fong,
>>
>> A vector of length 40503 is definitely not too large.  The code that
>> you included looks fine.  Is it possible that in the full version of the
>> script the variable "ps" is defined outside of the foreach loop?
>> It would be automatically exported in that case, and if it was large
>> enough, it could cause the serialization error.  If that is the case,
>> you could fix the problem by using the foreach argument
>> ".noexport='ps'".
>>
>> If that isn't the case, could you post a complete, self-contained
>> example that demonstrates the problem?
>>
>> - Steve
>>
>>
>> On Tue, Aug 10, 2010 at 4:07 PM, Fong Chun Chan
>> <fongchun at interchange.ubc.ca> wrote:
>> > Hi all,
>> >
>> > I am getting this serialization error when I am trying to run a simple
>> > parallel process script.  I just want to test if it works.
>> >
>> > foreach (i = 1:length(U133_ps), .combine="rbind") %dopar% {
>> > + ps <- U133_ps[i]
>> > + ps
>> > + }
>> > Error: serialization is too large to store in a raw vector
>> >
>> > U133_ps is a single column vector of length 40503.  Is this too large of
>> > a
>> > dataset to iterate over?  If not, how can I get around this problem?
>> >
>> > Thanks,
>> >
>> > Fong
>> >
>> >        [[alternative HTML version deleted]]
>> >
>> > _______________________________________________
>> > R-sig-hpc mailing list
>> > R-sig-hpc at r-project.org
>> > https://stat.ethz.ch/mailman/listinfo/r-sig-hpc
>> >
>>
>> _______________________________________________
>> R-sig-hpc mailing list
>> R-sig-hpc at r-project.org
>> https://stat.ethz.ch/mailman/listinfo/r-sig-hpc
>
>