{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "%matplotlib inline"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "\n# Visualization of MLP weights on MNIST\n\n\nSometimes looking at the learned coefficients of a neural network can provide\ninsight into the learning behavior. For example if weights look unstructured,\nmaybe some were not used at all, or if very large coefficients exist, maybe\nregularization was too low or the learning rate too high.\n\nThis example shows how to plot some of the first layer weights in a\nMLPClassifier trained on the MNIST dataset.\n\nThe input data consists of 28x28 pixel handwritten digits, leading to 784\nfeatures in the dataset. Therefore the first layer weight matrix have the shape\n(784, hidden_layer_sizes[0]).  We can therefore visualize a single column of\nthe weight matrix as a 28x28 pixel image.\n\nTo make the example run faster, we use very few hidden units, and train only\nfor a very short time. Training longer would result in weights with a much\nsmoother spatial appearance.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "import matplotlib.pyplot as plt\nfrom sklearn.datasets import fetch_openml\nfrom sklearn.neural_network import MLPClassifier\n\nprint(__doc__)\n\n# Load data from https://www.openml.org/d/554\nX, y = fetch_openml('mnist_784', version=1, return_X_y=True)\nX = X / 255.\n\n# rescale the data, use the traditional train/test split\nX_train, X_test = X[:60000], X[60000:]\ny_train, y_test = y[:60000], y[60000:]\n\n# mlp = MLPClassifier(hidden_layer_sizes=(100, 100), max_iter=400, alpha=1e-4,\n#                     solver='sgd', verbose=10, tol=1e-4, random_state=1)\nmlp = MLPClassifier(hidden_layer_sizes=(50,), max_iter=10, alpha=1e-4,\n                    solver='sgd', verbose=10, tol=1e-4, random_state=1,\n                    learning_rate_init=.1)\n\nmlp.fit(X_train, y_train)\nprint(\"Training set score: %f\" % mlp.score(X_train, y_train))\nprint(\"Test set score: %f\" % mlp.score(X_test, y_test))\n\nfig, axes = plt.subplots(4, 4)\n# use global min / max to ensure all weights are shown on the same scale\nvmin, vmax = mlp.coefs_[0].min(), mlp.coefs_[0].max()\nfor coef, ax in zip(mlp.coefs_[0].T, axes.ravel()):\n    ax.matshow(coef.reshape(28, 28), cmap=plt.cm.gray, vmin=.5 * vmin,\n               vmax=.5 * vmax)\n    ax.set_xticks(())\n    ax.set_yticks(())\n\nplt.show()"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.7"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}