保留正文与补采附件。源页面当前未重新核验；具体限制见交付清单。

Revisit consent button We value your privacy We use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking "Accept All", you consent to our use of cookies. Customise Reject All Accept All Customise Consent Preferences We use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below. The cookies that are categorised as "Necessary" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more NecessaryAlways Active Necessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.

Functional Functional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.

Analytics Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.

Performance Performance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.

Advertisement Advertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.

Uncategorised Other uncategorised cookies are those that are being analysed and have not been classified into a category as yet. No cookies to display. Reject All Save My Preferences Accept All Toggle SearchToggle Mobile Navigation Publish AI, ML & data-science insights to a global community of data professionals. Sign inSubmit an Article LatestEditor's PicksNewsletterShipAI

Write For TDS LinkedInXToggle Search Machine Learning

Classifying Handwritten Digits Using A Multilayer Perceptron Classifier (MLP)

What is a Multilayer Perceptron? What are the pros and cons of MLP? Can we classify handwritten digits accurately using a MLP classifier... Serafeim Loukas, PhD November 27, 20217 min read

What is a Multilayer Perceptron? What are the pros and cons of MLP? Can we classify handwritten digits accurately using a MLP classifier? How do learnt weights look like?

Figure 1: A Multilayer Perceptron Network source. Figure 1: A Multilayer Perceptron Network (source). Learn this step by step with the interactive Machine Learning roadmap.

1. Short Introduction

1.1 What is a Multilayer Perceptron (MLP)?

An MLP is a supervised machine learning (ML) algorithm that belongs in the class of feedforward artificial neural networks [1]. The algorithm essentially is trained on the data in order to learn a function. Given a set of features and a target variable (e.g. labels) it learns a non-linear function for either classification or regression. In this article, we will only focus on the classification case.

1.2 Is there any similarity between the MLP and the Logistic Regression?

There is! The logistic regression has only two layers i.e. the input and output however, in the case of a MLP model, the only difference is that we can have additional intermediate nonlinear layers. These are known as hidden layers. Except for the input nodes (nodes that belong to the input layer), each node is a neuron that uses a nonlinear activation function [1]. Due to this non-linearity nature, the MLP can learn complex nonlinear functions and thus, distinguish data that are not linearly separable! See Figure 2 below for a visual representation of an MLP classifier with one hidden layer.

NEW : After a great deal of hard work and staying behind the scenes for quite a while, we're excited to now offer our expertise through a platform, the "Data Science Hub" on Patreon (https://www.patreon.com/TheDataScienceHub). This hub is our way of providing you with bespoke consulting services and comprehensive responses to all your inquiries , ranging from Machine Learning to strategic data analytics planning.

1.3 How is an MLP trained?

MLP uses backpropagation for training [1]. You can have a look at this website here for the formal mathematical formulation.

1.4 Main Advantages and Disadvantages of MLP

Pros

Cons

Figure 2: A MLP with one hidden layer and with a scalar output. Image adapted from scikit-learn python documentation. Figure 2: A MLP with one hidden layer and with a scalar output. Image adapted from scikit-learn python documentation.

2. Python hands-on example using scikit-learn

2.1 The dataset

For this hands-on example we will use the MNIST dataset. The MNIST database is famous database of handwritten digits that is used for training several ML models [5]. There are handwritten images of 10 different digits so the number of classes is **10**(see Figure 3). Note : Since we deal with images , these are represented by 2D arrays and the initial dimension of the data is 28** by **28 for each image (28x28 pixels). The 2D images are then flattened and thus, represented by vectors at the end. Each 2D image is transformed into a 1D vector with dimension [1, 28x28] = **[1, 784]. Finally, our dataset has _784 features/variables/columns.**_ Figure 3: Some samples from the dataset Source. Figure 3: Some samples from the dataset (Source).

2.2 Data import and preparation

text

import matplotlib.pyplot as plt
from sklearn.datasets import fetch_openml
from sklearn.neural_network import MLPClassifier

# Load data
X, y = fetch_openml("mnist_784", version=1, return_X_y=True)

text

# Normalize intensity of images to make it in the range [0,1] since 255 is the max (white).
X = X / 255.0

Remember that each 2D image is now transformed into a 1D vector with dimension [1, 28x28] = **[1, 784].** Let's verify this now. text

print(X.shape)

This returns: **(70000, 784)**. We have 70k flattend images (samples), each containing 784 pixels (28*28=784) (variables/features). Thus, the input layer weight matrix will have shape 784 x #neurons_in_1st_hidden_layer. The output layer weight matrix will have shape #neurons_in_3rd_hidden_layer x #number_of_classes.

2.3 Model training

Now let's build the model, train it and perform the classification. We will use 3 hidden layers with 50,20 and 10 neurons each, respectively. Also, we will set the max iterations to 100 and the learning rate to 0.1. These are the hyperparameters that I mentioned in the Introduction. We will not fine-tune them here. text

# Split the data into train/test sets
X_train, X_test = X[:60000], X[60000:]
y_train, y_test = y[:60000], y[60000:]

classifier = MLPClassifier(
    hidden_layer_sizes=(50,20,10),
    max_iter=100,
    alpha=1e-4,
    solver="sgd",
    verbose=10,
    random_state=1,
    learning_rate_init=0.1,
)

text

# fit the model on the training data
classifier.fit(X_train, y_train)

2.4 Model evaluation

Now, let's evaluate the model. We will estimate the mean accuracy on the training and test data and labels. text

print("Training set score: %f" % classifier.score(X_train, y_train))
print("Test set score: %f" % classifier.score(X_test, y_test))

Training set score: 0.998633 Test set score: 0.970300 Great results!

2.5 Visualizing the Cost Function evolution

How fast was the loss decreased during training? Let's make a nice plot! text

fig, axes = plt.subplots(1, 1)

text

axes.plot(classifier.loss_curve_, 'o-')
axes.set_xlabel("number of iteration")
axes.set_ylabel("loss")
plt.show()

Figure 4: The evolution of the loss across the training iterations. Figure made by the author. Figure 4: The evolution of the loss across the training iterations. Figure made by the author. Here we see that the loss decreases really fast during training and it saturates after the 40th iteration (remember that we defined max 100 iterations as hyperparameter).

2.6 Visualizing the learnt weights

Here we need first to understand how the weights (learnt model parameters of each layer) are stored. Based on the documentation, the attribute classifier.coefs_ is a list of shape (n_layers-1, )of weight arrays, where weight matrix at index i represents the weights between layer i and layer i+1. In this example, we defined 3 hidden layers and we also have the input and output layers. Thus, **** we expect to have 4 weight arrays for the between-layers weights (**in-L1, L1-L2, L2-L3**and**L2-out**in Figure 5). Similarly, classifier.intercepts_is a list of bias vectors, where the vector at index i represents the bias values added to layer i+1. Figure 5: Handmade figure by the author on Notes iOS. Figure 5: Handmade figure by the author on Notes (iOS). Let's verify this: text

len(classifier.intercepts_) == len(classifier.coefs_) == 4

correctly returns True. Reminder: The input layer weight matrix will have shape 784 x #neurons_in_1st_hidden_layer. The output layer weight matrix will have shape #neurons_in_3rd_hidden_layer x #number_of_classes.

Visualizing the learnt weights of the input layer

text

target_layer = 0 #0 is input, 1 is 1st hidden etc

text

fig, axes = plt.subplots(1, 1, figsize=(15,6))
axes.imshow(np.transpose(classifier.coefs_[target_layer]), cmap=plt.get_cmap("gray"), aspect="auto")

text

axes.set_xlabel(f"number of neurons in {target_layer}")
axes.set_ylabel("neurons in output layer")
plt.show()

Figure 6: Visualization of the learnt weights of the neurons between the input and the 1st hidden layer. Figure made by the author. Figure 6: Visualization of the learnt weights of the neurons between the input and the 1st hidden layer. Figure made by the author. Or reshape and plot them as 2D images again. text

# choose layer to plot
target_layer = 0 #0 is input, 1 is 1st hidden etc

text

fig, axes = plt.subplots(4, 4)
vmin, vmax = classifier.coefs_[0].min(), classifier.coefs_[target_layer].max()

text

for coef, ax in zip(classifier.coefs_[0].T, axes.ravel()):
    ax.matshow(coef.reshape(28, 28), cmap=plt.cm.gray, vmin=0.5 * vmin, vmax=0.5 * vmax)
    ax.set_xticks(())
    ax.set_yticks(())
plt.show()

3. Summary

MLP classifier is a very powerful neural network model that enables the learning of non-linear functions for complex data. The method uses forward propagation to build the weights and then it computes the loss. Next, back propagation is used to update the weights so that the loss is reduced. This is done in an iterative way and the number of iterations is an input hyperparameter as I explained in the Introduction. Other important hyperparameters are the number of neurons in each hidden layer and the number of hidden layers in total. These need to be fine-tuned.

That's all folks! Hope you liked this article!

Subscribe to my mailing list in just 5 seconds: https://seralouk.medium.com/subscribe

Stay tuned & support this effort

If you liked and found this article useful, follow me! Questions? Post them as a comment and I will reply as soon as possible.

Latest posts

How to use Python & SQL to manipulate data in 1 min Time-Series Forecasting: Predicting Stock Prices Using Facebook's Prophet Model ROC Curve Explained using a COVID-19 hypothetical example: Binary & Multi-Class Classification... Support Vector Machines (SVM) clearly explained: A python tutorial for classification problems... Everything you need to know about Min-Max normalization in Python How Scikit-Learn's StandardScaler works

References

[1] https://en.wikipedia.org/wiki/Multilayer_perceptron [2] https://scikit-learn.org/stable/modules/neural_networks_supervised.html#mlp-tips [3] https://scikit-learn.org/stable/modules/neural_networks_supervised.html#mathematical-formulation [4] https://en.wikipedia.org/wiki/Backpropagation [5] https://en.wikipedia.org/wiki/MNIST_database

Get in touch with me

Register Now to Continue Reading

Unlock Towards Data Science with a Free Account

All articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more. Register for freeSign in to your account

Written By

Serafeim Loukas, PhD

See all from Serafeim Loukas, PhD ClassificationMachine LearningMultilayer PerceptronNeural NetworksPerceptron Share This Article

Towards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program. Write for TDS

Related Articles

Consistent Semi-Supervised, Explainable Multi-Tasking for Medical Imaging

Artificial Intelligence MultiMix: Sparingly Supervised, Extreme Multitask Learning from Medical Images Ayaan Haque June 9, 202217 min read

Improve tabular data prediction with Large Language Model through OpenAI API

Large Language Models Python implementation with machine learning classification, prompt engineering, feature engineering on text embedding and model... Tony Zhang July 13, 202311 min read

Why Are Convolutional Neural Networks Great For Images?

Deep Learning How data symmetry informs neural network architectures Caroline Arnold April 30, 20253 min read

Fast Anomaly Detection in Images With Python

Machine Learning How I improved speed by 10x Anthony Cavin June 28, 20227 min read

The Power of Ensemble Methods in Machine Learning

Artificial Intelligence How to boost the performance with bagging and boosting Soner Yıldırım June 6, 20205 min read

Interactive Distribution Plots with Plotly

Artificial Intelligence How to create informative distribution plots Soner Yıldırım May 29, 20204 min read

Essential Questions to Ask Before Starting a Data Science Project

Artificial Intelligence How to make sure your data science project gets a good start Shahwaiz January 7, 20226 min read

Don't Worry, Excel is Surprisingly Effective

Machine Learning Why you shouldn't be ashamed to use Excel as a data scientist Tyler Folkman March 4, 20204 min read

Can Transformers Learn to Strategize?

Deep Learning TicTacGPT for playing simple board games Charlie O'Neill September 8, 202330 min read

Your home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals. Subscribe to Our NewsletterWrite For TDS•About•Advertise•Privacy Policy•Terms of Use © Insight Media Group, LLC 2026•The New Stack•roadmap.sh•Creator Network

Recommended Articles

Close