[R] gamm problem/error fitting smooth by factor

[Mathew Guilfoyle]

I have a (unbalanced) dataset of time series collected across several subjects (n~500, ~60000 observations).  I would like to model the overall smooth time trend of a variable and how this trend differs by various categorical factors, with the subject as a random effect.

My baseline model 

m1 = gamm(v1 ~ s(time), random=list(id=~1+time), data=dat)

shows that time is a significant term.

I have then tried to run this model:

m2 = gamm(v1 ~ s(time)+s(time, by=fac1), random=list(id=~1+time), data=dat)

where fac1 is a binary factor.

My intuitive understanding of this is that the first smooth term will capture the overall major trend common to both groups of fac1 and the second smooth will model the deviation from this mean trend for the fac1==1 subgroup.  However, I get the error:

 Error in MEestimate(lmeSt, grps) : 
  Singularity in backsolve at level 0, block 1 

I've tried other models to isolate the problem but get the exact same error

m3 = gamm(v1 ~ s(time)+s(time, by=fac1), random=list(id=~1), data=dat)   #remove the time random effect
m4 = gamm(v1 ~ fac1+s(time)+s(time, by=fac1), random=list(id=~1+time), data=dat)   #have the factor as a main effect also

I'm not sure if the whole notion of what I'm trying to do is wrong-headed or if I need to adjust some parameters to get the model (m2) to fit.

Thanks