[R] glm predict issue

[Weidong Gu]

Hi,

This might be due to the fact that factor levels are arbitary unless
they are ordinal, even that quantitative relationships between levels
are unclear. Therefore, the model has no way to predict unseen factor
levels.

Does it make sense to treat 'No_databases' as numeric instead of a
factor variable?

Weidong

On Mon, Dec 26, 2011 at 6:29 AM, Giovanni Azua <bravegag at gmail.com> wrote:
> Hello,
>
> I have tried reading the documentation and googling for the answer but reviewing the online matches I end up more confused than before.
>
> My problem is apparently simple. I fit a glm model (2^k experiment), and then I would like to predict the response variable (Throughput) for unseen factor levels.
>
> When I try to predict I get the following error:
>> throughput.pred <- predict(throughput.fit,experiments,type="response")
> Error in model.frame.default(Terms, newdata, na.action = na.action, xlev = object$xlevels) :
>  factor 'No_databases' has new level(s) 200, 400, 600, 800, 1000
>
> Of course these are new factor levels, it is exactly what I am trying to achieve i.e. extrapolate the values of Throughput.
>
> Can anyone please advice? Below I include all details.
>
> Thanks in advance,
> Best regards,
> Giovanni
>
>> # define the extreme (factors and levels)
>> experiments <- expand.grid(No_databases   = seq(1000,100,by=-200),
> +                                                  Partitioning   = c("sharding", "replication"),
> +                                                  No_middlewares = seq(500,100,by=-100),
> +                                                  Queue_size     = c(100))
>> experiments$No_databases <- as.factor(experiments$No_databases)
>> experiments$Partitioning <- as.factor(experiments$Partitioning)
>> experiments$No_middlewares <- as.factor(experiments$No_middlewares)
>> experiments$Queue_size <- as.factor(experiments$Queue_size)
>> str(experiments)
> 'data.frame':   50 obs. of  4 variables:
>  $ No_databases  : Factor w/ 5 levels "200","400","600",..: 5 4 3 2 1 5 4 3 2 1 ...
>  $ Partitioning  : Factor w/ 2 levels "sharding","replication": 1 1 1 1 1 2 2 2 2 2 ...
>  $ No_middlewares: Factor w/ 5 levels "100","200","300",..: 5 5 5 5 5 5 5 5 5 5 ...
>  $ Queue_size    : Factor w/ 1 level "100": 1 1 1 1 1 1 1 1 1 1 ...
>  - attr(*, "out.attrs")=List of 2
>  ..$ dim     : Named int  5 2 5 1
>  .. ..- attr(*, "names")= chr  "No_databases" "Partitioning" "No_middlewares" "Queue_size"
>  ..$ dimnames:List of 4
>  .. ..$ No_databases  : chr  "No_databases=1000" "No_databases= 800" "No_databases= 600" "No_databases= 400" ...
>  .. ..$ Partitioning  : chr  "Partitioning=sharding" "Partitioning=replication"
>  .. ..$ No_middlewares: chr  "No_middlewares=500" "No_middlewares=400" "No_middlewares=300" "No_middlewares=200" ...
>  .. ..$ Queue_size    : chr "Queue_size=100"
>> head(experiments)
>  No_databases Partitioning No_middlewares Queue_size
> 1         1000     sharding            500        100
> 2          800     sharding            500        100
> 3          600     sharding            500        100
> 4          400     sharding            500        100
> 5          200     sharding            500        100
> 6         1000  replication            500        100
>> # or
>> throughput.fit <- glm(log(Throughput)~(No_databases*No_middlewares)+Partitioning+Queue_size,
> +                                       data=throughput)
>> summary(throughput.fit)
>
> Call:
> glm(formula = log(Throughput) ~ (No_databases * No_middlewares) +
>    Partitioning + Queue_size, data = throughput)
>
> Deviance Residuals:
>    Min       1Q   Median       3Q      Max
> -2.5966  -0.6612  -0.1944   0.5548   3.2136
>
> Coefficients:
>                              Estimate Std. Error t value Pr(>|t|)
> (Intercept)                    5.74701    0.09127  62.970  < 2e-16 ***
> No_databases4                  0.43309    0.10985   3.943 8.66e-05 ***
> No_middlewares2               -1.99374    0.11035 -18.067  < 2e-16 ***
> No_middlewares4               -1.23004    0.10969 -11.214  < 2e-16 ***
> Partitioningreplication        0.33291    0.06181   5.386 9.15e-08 ***
> Queue_size100                  0.15850    0.06181   2.564   0.0105 *
> No_databases4:No_middlewares2  2.71525    0.15262  17.791  < 2e-16 ***
> No_databases4:No_middlewares4  1.94191    0.15226  12.754  < 2e-16 ***
> ---
> Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
>
> (Dispersion parameter for gaussian family taken to be 0.8921778)
>
>    Null deviance: 2175.58  on 936  degrees of freedom
> Residual deviance:  828.83  on 929  degrees of freedom
> AIC: 2562.2
>
> Number of Fisher Scoring iterations: 2
>
>> throughput.pred <- predict(throughput.fit,experiments,type="response")
> Error in model.frame.default(Terms, newdata, na.action = na.action, xlev = object$xlevels) :
>  factor 'No_databases' has new level(s) 200, 400, 600, 800, 1000
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.