Pau,

Thanks a lot for your email, I found it very helpful. Please see below for
my reply, thanks.

-Jack

On Wed, Jul 14, 2010 at 10:36 AM, Pau Carrio Gaspar <paucarrio@gmail.com>wrote:

>  Hello Jack,
>
> 1 ) why do you thought that " larger C is prone to overfitting than smaller
> C" ?
>

  *There is some statement in the link http://www.dtreg.com/svm.htm

"To allow some flexibility in separating the categories, SVM models have a
cost parameter, C, that controls the trade off between allowing training
errors and forcing rigid margins. It   creates a soft margin that permits
some misclassifications. Increasing the value of C increases the cost of
misclassifying points and forces the creation of a more accurate model that
may not generalize well."

My understanding is that this means larger C may not generalize well (prone
to overfitting).
*
2 ) if you look at the formulation of the quadratic program problem you will
see that  C rules the error of the "cutting plane " ( and overfitting ).
Therfore for hight C you allow that the "cutting plane" cuts worse the set,
so SVM needs less points to build it. a proper explanation is in Kristin P.
Bennett and Colin Campbell, "Support Vector Machines: Hype or Hallelujah?",
SIGKDD Explorations, 2,2, 2000, 1-13.
http://www.idi.ntnu.no/emner/it3704/lectures/papers/Bennett_2000_Support.pdf

*Could you be more specific about this? I don't quite understand. *

>
> 3) you might find usefull this plots:
>
> library(e1071)
> m1 <- matrix( c(
> 0,    0,    0,    1,    1,    2,     1, 2,    3,    2,    3, 3, 0,
> 1,2,3,    0, 1, 2, 3,
> 1,    2,    3,    2,    3,    3,     0, 0,    0,    1, 1, 2, 4, 4,4,4,
> 0, 1, 2, 3,
> 1,    1,    1,    1,    1,    1,    -1,-1,  -1,-1,-1,-1, 1 ,1,1,1,     1,
> 1,-1,-1
> ), ncol = 3 )
>
> Y = m1[,3]
> X = m1[,1:2]
>
> df = data.frame( X , Y )
>
> par(mfcol=c(4,2))
> for( cost in c( 1e-3 ,1e-2 ,1e-1, 1e0,  1e+1, 1e+2 ,1e+3)) {
> #cost <- 1
> model.svm <- svm( Y ~ . , data = df ,  type = "C-classification" , kernel =
> "linear", cost = cost,
>                          scale =FALSE )
> #print(model.svm$SV)
>
> plot(x=0,ylim=c(0,5), xlim=c(0,3),main= paste( "cost: ",cost, "#SV: ",
> nrow(model.svm$SV) ))
> points(m1[m1[,3]>0,1], m1[m1[,3]>0,2], pch=3, col="green")
> points(m1[m1[,3]<0,1], m1[m1[,3]<0,2], pch=4, col="blue")
> points(model.svm$SV[,1],model.svm$SV[,2], pch=18 , col = "red")
> }
> *
> *

*Thanks a lot for the code, I really appreciate it. I've run it, but I am
not sure how should I interpret the scatter plot, although it is obvious
that number of SVs decreases with cost increasing. *

>
> Regards
> Pau
>
>
> 2010/7/14 Jack Luo <jluo.rhelp@gmail.com>
>
>> Hi,
>>
>> I have a question about the parameter C (cost) in svm function in e1071. I
>> thought larger C is prone to overfitting than smaller C, and hence leads
>> to
>> more support vectors. However, using the Wisconsin breast cancer example
>> on
>> the link:
>> http://planatscher.net/svmtut/svmtut.html
>> I found that the largest cost have fewest support vectors, which is
>> contrary
>> to what I think. please see the scripts below:
>> Am I misunderstanding something here?
>>
>> Thanks a bunch,
>>
>> -Jack
>>
>> > model1 <- svm(databctrain, classesbctrain, kernel = "linear", cost =
>> 0.01)
>> > model2 <- svm(databctrain, classesbctrain, kernel = "linear", cost = 1)
>> > model3 <- svm(databctrain, classesbctrain, kernel = "linear", cost =
>> 100)
>> > model1
>>
>> Call:
>> svm.default(x = databctrain, y = classesbctrain, kernel = "linear",
>>    cost = 0.01)
>>
>>
>> Parameters:
>>   SVM-Type:  C-classification
>>  SVM-Kernel:  linear
>>       cost:  0.01
>>      gamma:  0.1111111
>>
>> Number of Support Vectors:  99
>>
>> > model2
>>
>> Call:
>> svm.default(x = databctrain, y = classesbctrain, kernel = "linear",
>>    cost = 1)
>>
>>
>> Parameters:
>>   SVM-Type:  C-classification
>>  SVM-Kernel:  linear
>>       cost:  1
>>      gamma:  0.1111111
>>
>> Number of Support Vectors:  46
>>
>> > model3
>>
>> Call:
>> svm.default(x = databctrain, y = classesbctrain, kernel = "linear",
>>    cost = 100)
>>
>>
>> Parameters:
>>   SVM-Type:  C-classification
>>  SVM-Kernel:  linear
>>       cost:  100
>>      gamma:  0.1111111
>>
>> Number of Support Vectors:  44
>>
>>        [[alternative HTML version deleted]]
>>
>> ______________________________________________
>> R-help@r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-help
>> PLEASE do read the posting guide
>> http://www.R-project.org/posting-guide.html
>> and provide commented, minimal, self-contained, reproducible code.
>>
>
>

	[[alternative HTML version deleted]]

