{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "%matplotlib inline"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "\n# Plot Ridge coefficients as a function of the L2 regularization\n\n\n.. currentmodule:: sklearn.linear_model\n\n:class:`Ridge` Regression is the estimator used in this example.\nEach color in the left plot represents one different dimension of the\ncoefficient vector, and this is displayed as a function of the\nregularization parameter. The right plot shows how exact the solution\nis. This example illustrates how a well defined solution is\nfound by Ridge regression and how regularization affects the\ncoefficients and their values. The plot on the right shows how\nthe difference of the coefficients from the estimator changes\nas a function of regularization.\n\nIn this example the dependent variable Y is set as a function\nof the input features: y = X*w + c. The coefficient vector w is\nrandomly sampled from a normal distribution, whereas the bias term c is\nset to a constant.\n\nAs alpha tends toward zero the coefficients found by Ridge\nregression stabilize towards the randomly sampled vector w.\nFor big alpha (strong regularisation) the coefficients\nare smaller (eventually converging at 0) leading to a\nsimpler and biased solution.\nThese dependencies can be observed on the left plot.\n\nThe right plot shows the mean squared error between the\ncoefficients found by the model and the chosen vector w.\nLess regularised models retrieve the exact\ncoefficients (error is equal to 0), stronger regularised\nmodels increase the error.\n\nPlease note that in this example the data is non-noisy, hence\nit is possible to extract the exact coefficients.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "# Author: Kornel Kielczewski -- <kornel.k@plusnet.pl>\n\nprint(__doc__)\n\nimport matplotlib.pyplot as plt\nimport numpy as np\n\nfrom sklearn.datasets import make_regression\nfrom sklearn.linear_model import Ridge\nfrom sklearn.metrics import mean_squared_error\n\nclf = Ridge()\n\nX, y, w = make_regression(n_samples=10, n_features=10, coef=True,\n                          random_state=1, bias=3.5)\n\ncoefs = []\nerrors = []\n\nalphas = np.logspace(-6, 6, 200)\n\n# Train the model with different regularisation strengths\nfor a in alphas:\n    clf.set_params(alpha=a)\n    clf.fit(X, y)\n    coefs.append(clf.coef_)\n    errors.append(mean_squared_error(clf.coef_, w))\n\n# Display results\nplt.figure(figsize=(20, 6))\n\nplt.subplot(121)\nax = plt.gca()\nax.plot(alphas, coefs)\nax.set_xscale('log')\nplt.xlabel('alpha')\nplt.ylabel('weights')\nplt.title('Ridge coefficients as a function of the regularization')\nplt.axis('tight')\n\nplt.subplot(122)\nax = plt.gca()\nax.plot(alphas, errors)\nax.set_xscale('log')\nplt.xlabel('alpha')\nplt.ylabel('error')\nplt.title('Coefficient error as a function of the regularization')\nplt.axis('tight')\n\nplt.show()"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.7"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}