I started watching AN’s coursera course on Machine learning, a field I want to get into hopefully in the near future. The clarity of Andrew’s presentation kept me watching one video after another. His example of quadratic versus quartic model for housing prices provided good motivation for penalization terms in regression models to reduce overfitting. But when introducing a general shrinkage term of the form , where ‘s are the polynomial regression coefficients, he seems to suggest it’s very complicated to understand how keeping the parameters small yield simpler models. Indeed penalizing all the coefficients is different from just a subset of preselected ones. Here is how I would explain this phenomenon in the case of polynomial regression: given a set of say 10 points , that roughly look like a quadratic curve, one can of course fit a 9th degree polynomial to match all 10 points. But doing so would require very large coefficients because the points are really not growing as fast as a typical 9th degree polynomial, thus one requires a lot of cancellation in polynomial terms to create something that locally looks like a small perturbation of a quadratic. This easily increases the total absolute value size of all the coefficients, or . For those who have not seen this phenomenon in analytical estimates: consider the power series for . The individual coefficients are quite big, especially for small degree as I chose the intentionally vicious multiplier , whose powers are not easily dominated by the factorial denominators. But we know well that . The reason is simply cancellation of plus and minus terms. Now to go a bit further, if we are just interested in approximating for to reasonable precision, we don’t need an infinite series. In fact we don’t even need an degree polynomial. This problem is exactly scalable from that of approximating for . I would probably choose a cubic, that is, . Presumably Tichnoff regularization would let us do that.

Recent Posts
Recent Comments
Simon on How much it costs to raise a… aquazorcarson on How much it costs to raise a… Simon on How much it costs to raise a… aquazorcarson on Friendship is costly Erwin on Friendship is costly Archives
 July 2017
 June 2017
 October 2016
 September 2016
 July 2016
 June 2016
 May 2016
 March 2016
 July 2015
 May 2015
 March 2015
 February 2015
 January 2015
 November 2014
 October 2014
 June 2014
 March 2014
 February 2014
 December 2013
 November 2013
 October 2013
 August 2013
 July 2013
 May 2013
 April 2013
 March 2013
 January 2013
 September 2012
 January 2012
 December 2011
 September 2011
 August 2011
 July 2011
 June 2011
 April 2011
 March 2011
 February 2011
 January 2011
 November 2010
 October 2010
 September 2010
 July 2010
 June 2010
 May 2010
 April 2010
 March 2010
 February 2010
 January 2010
 December 2009
 October 2009
 September 2009
 August 2009
 July 2009
 June 2009
 May 2009
 April 2009
 March 2009
 February 2009
 January 2009
 October 2008
 September 2008
 July 2008
 June 2008
 May 2008
 April 2008
 March 2008
 February 2008
 January 2008
 October 2007
 September 2007
 August 2007
 July 2007
 April 2007
 March 2007
 February 2007
 December 2006
 November 2006
 October 2006
 September 2006
 August 2006
Categories
Meta