{
  "source_file": "0930",
  "source_platform": "技术平台",
  "author": "Jason Brownlee / Machine Learning Mastery",
  "title_listed": "How to Develop a CNN for Image Classification",
  "keywords": "CNN,图像分类,教程",
  "domain": "图像处理",
  "content_type": "教程",
  "is_paid_flag": "否",
  "url": "https://machinelearningmastery.com/how-to-develop-a-cnn-from-scratch-for-cifar-10-photo-classification/",
  "publish_date": "2019-05",
  "corpus_stage": "预训练/SFT",
  "notes": "CNN图像分类教程",
  "id": "6dda3c1e3103",
  "collected_at": "2026-10-02T17:54:50",
  "title": "How to Develop a CNN From Scratch for CIFAR-10 Photo Classification - MachineLearningMastery.com",
  "body_markdown": "How to Develop a CNN From Scratch for CIFAR-10 Photo Classification\nBy\nJason Brownlee\non\nAugust 28, 2020\nin\nDeep Learning for Computer Vision\n120\nShare\nPost\nShare\nDiscover how to develop a deep convolutional neural network model from scratch for the CIFAR-10 object classification dataset.\nThe CIFAR-10 small photo classification problem is a standard dataset used in computer vision and deep learning.\nAlthough the dataset is effectively solved, it can be used as the basis for learning and practicing how to develop, evaluate, and use convolutional deep learning neural networks for image classification from scratch.\nThis includes how to develop a robust test harness for estimating the performance of the model, how to explore improvements to the model, and how to save the model and later load it to make predictions on new data.\nIn this tutorial, you will discover how to develop a convolutional neural network model from scratch for object photo classification.\nAfter completing this tutorial, you will know:\nHow to develop a test harness to develop a robust evaluation of a model and establish a baseline of performance for a classification task.\nHow to explore extensions to a baseline model to improve learning and model capacity.\nHow to develop a finalized model, evaluate the performance of the final model, and use it to make predictions on new images.\nKick-start your project\nwith my new book\nDeep Learning for Computer Vision\n, including\nstep-by-step tutorials\nand the\nPython source code\nfiles for all examples.\nLet’s get started.\nUpdated Oct/2019\n: Updated for Keras 2.3 and TensorFlow 2.0.\nHow to Develop a Convolutional Neural Network From Scratch for CIFAR-10 Photo Classification\nPhoto by\nRose Dlhopolsky\n, some rights reserved.\nTutorial Overview\nThis tutorial is divided into six parts; they are:\nCIFAR-10 Photo Classification Dataset\nModel Evaluation Test Harness\nHow to Develop a Baseline Model\nHow to Develop an Improved Model\nHow to Develop Further Improvements\nHow to Finalize the Model and Make Predictions\nWant Results with Deep Learning for Computer Vision?\nTake my free 7-day email crash course now (with sample code).\nClick to sign-up and also get a free PDF Ebook version of the course.\nClick here to subscribe\nCIFAR-10 Photo Classification Dataset\nCIFAR is an acronym that stands for the\nCanadian Institute For Advanced Research\nand the\nCIFAR-10 dataset\nwas developed along with the CIFAR-100 dataset by researchers at the CIFAR institute.\nThe dataset is comprised of 60,000 32×32 pixel color photographs of objects from 10 classes, such as frogs, birds, cats, ships, etc. The class labels and their standard associated integer values are listed below.\n0: airplane\n1: automobile\n2: bird\n3: cat\n4: deer\n5: dog\n6: frog\n7: horse\n8: ship\n9: truck\nThese are very small images, much smaller than a typical photograph, and the dataset was intended for computer vision research.\nCIFAR-10 is a well-understood dataset and widely used for benchmarking computer vision algorithms in the field of machine learning. The problem is “\nsolved\n.” It is relatively straightforward to achieve 80% classification accuracy. Top performance on the problem is achieved by deep learning convolutional neural networks with a classification accuracy above 90% on the test dataset.\nThe example below loads the CIFAR-10 dataset using the Keras API and creates a plot of the first nine images in the training dataset.\n# example of loading the cifar10 dataset\nfrom matplotlib import pyplot\nfrom keras.datasets import cifar10\n# load dataset\n(trainX, trainy), (testX, testy) = cifar10.load_data()\n# summarize loaded dataset\nprint('Train: X=%s, y=%s' % (trainX.shape, trainy.shape))\nprint('Test: X=%s, y=%s' % (testX.shape, testy.shape))\n# plot first few images\nfor i in range(9):\n\t# define subplot\n\tpyplot.subplot(330 + 1 + i)\n\t# plot raw pixel data\n\tpyplot.imshow(trainX[i])\n# show the figure\npyplot.show()\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n14\n15\n16\n# example of loading the cifar10 dataset\nfrom\nmatplotlib\nimport\npyplot\nfrom\nkeras\n.\ndatasets\nimport\ncifar10\n# load dataset\n(\ntrainX\n,\ntrainy\n)\n,\n(\ntestX\n,\ntesty\n)\n=\ncifar10\n.\nload_data\n(\n)\n# summarize loaded dataset\nprint\n(\n'Train: X=%s, y=%s'\n%\n(\ntrainX\n.\nshape\n,\ntrainy\n.\nshape\n)\n)\nprint\n(\n'Test: X=%s, y=%s'\n%\n(\ntestX\n.\nshape\n,\ntesty\n.\nshape\n)\n)\n# plot first few images\nfor\ni\nin\nrange\n(\n9\n)\n:\n# define subplot\npyplot\n.\nsubplot\n(\n330\n+\n1\n+\ni\n)\n# plot raw pixel data\npyplot\n.\nimshow\n(\ntrainX\n[\ni\n]\n)\n# show the figure\npyplot\n.\nshow\n(\n)\nRunning the example loads the CIFAR-10 train and test dataset and prints their shape.\nWe can see that there are 50,000 examples in the training dataset and 10,000 in the test dataset and that images are indeed square with 32×32 pixels and color, with three channels.\nTrain: X=(50000, 32, 32, 3), y=(50000, 1)\nTest: X=(10000, 32, 32, 3), y=(10000, 1)\n1\n2\nTrain: X=(50000, 32, 32, 3), y=(50000, 1)\nTest: X=(10000, 32, 32, 3), y=(10000, 1)\nA plot of the first nine images in the dataset is also created. It is clear that the images are indeed very small compared to modern photographs; it can be challenging to see what exactly is represented in some of the images given the extremely low resolution.\nThis low resolution is likely the cause of the limited performance that top-of-the-line algorithms are able to achieve on the dataset.\nPlot of a Subset of Images From the CIFAR-10 Dataset\nModel Evaluation Test Harness\nThe CIFAR-10 dataset can be a useful starting point for developing and practicing a methodology for solving image classification problems using convolutional neural networks.\nInstead of reviewing the literature on well-performing models on the dataset, we can develop a new model from scratch.\nThe dataset already has a well-defined train and test dataset that we will use. An alternative might be to perform k-fold cross-validation with a k=5 or k=10. This is desirable if there are sufficient resources. In this case, and in the interest of ensuring the examples in this tutorial execute in a reasonable time, we will not use k-fold cross-validation.\nThe design of the test harness is modular, and we can develop a separate function for each piece. This allows a given aspect of the test harness to be modified or interchanged, if we desire, separately from the rest.\nWe can develop this test harness with five key elements. They are the loading of the dataset, the preparation of the dataset, the definition of the model, the evaluation of the model, and the presentation of results.\nLoad Dataset\nWe know some things about the dataset.\nFor example, we know that the images are all pre-segmented (e.g. each image contains a single object), that the images all have the same square size of 32×32 pixels, and that the images are color. Therefore, we can load the images and use them for modeling almost immediately.\n# load dataset\n(trainX, trainY), (testX, testY) = cifar10.load_data()\n1\n2\n# load dataset\n(\ntrainX\n,\ntrainY\n)\n,\n(\ntestX\n,\ntestY\n)\n=\ncifar10\n.\nload_data\n(\n)\nWe also know that there are 10 classes and that classes are represented as unique integers.\nWe can, therefore, use a\none hot encoding\nfor the class element of each sample, transforming the integer into a 10 element binary vector with a 1 for the index of the class value. We can achieve this with the\nto_categorical()\nutility function.\n# one hot encode target values\ntrainY = to_categorical(trainY)\ntestY = to_categorical(testY)\n1\n2\n3\n# one hot encode target values\ntrainY\n=\nto_categorical\n(\ntrainY\n)\ntestY\n=\nto_categorical\n(\ntestY\n)\nThe\nload_dataset()\nfunction implements these behaviors and can be used to load the dataset.\n# load train and test dataset\ndef load_dataset():\n\t# load dataset\n\t(trainX, trainY), (testX, testY) = cifar10.load_data()\n\t# one hot encode target values\n\ttrainY = to_categorical(trainY)\n\ttestY = to_categorical(testY)\n\treturn trainX, trainY, testX, testY\n1\n2\n3\n4\n5\n6\n7\n8\n# load train and test dataset\ndef\nload_dataset\n(\n)\n:\n# load dataset\n(\ntrainX\n,\ntrainY\n)\n,\n(\ntestX\n,\ntestY\n)\n=\ncifar10\n.\nload_data\n(\n)\n# one hot encode target values\ntrainY\n=\nto_categorical\n(\ntrainY\n)\ntestY\n=\nto_categorical\n(\ntestY\n)\nreturn\ntrainX\n,\ntrainY\n,\ntestX\n,\ntestY\nPrepare Pixel Data\nWe know that the pixel values for each image in the dataset are unsigned integers in the range between no color and full color, or 0 and 255.\nWe do not know the best way to scale the pixel values for modeling, but we know that some scaling will be required.\nA good starting point is to normalize the pixel values, e.g. rescale them to the range [0,1]. This involves first converting the data type from unsigned integers to floats, then dividing the pixel values by the maximum value.\n# convert from integers to floats\ntrain_norm = train.astype('float32')\ntest_norm = test.astype('float32')\n# normalize to range 0-1\ntrain_norm = train_norm / 255.0\ntest_norm = test_norm / 255.0\n1\n2\n3\n4\n5\n6\n# convert from integers to floats\ntrain_norm\n=\ntrain\n.\nastype\n(\n'float32'\n)\ntest_norm\n=\ntest\n.\nastype\n(\n'float32'\n)\n# normalize to range 0-1\ntrain_norm\n=\ntrain_norm\n/\n255.0\ntest_norm\n=\ntest_norm\n/\n255.0\nThe\nprep_pixels()\nfunction below implement these behaviors and is provided with the pixel values for both the train and test datasets that will need to be scaled.\n# scale pixels\ndef prep_pixels(train, test):\n\t# convert from integers to floats\n\ttrain_norm = train.astype('float32')\n\ttest_norm = test.astype('float32')\n\t# normalize to range 0-1\n\ttrain_norm = train_norm / 255.0\n\ttest_norm = test_norm / 255.0\n\t# return normalized images\n\treturn train_norm, test_norm\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n# scale pixels\ndef\nprep_pixels\n(\ntrain\n,\ntest\n)\n:\n# convert from integers to floats\ntrain_norm\n=\ntrain\n.\nastype\n(\n'float32'\n)\ntest_norm\n=\ntest\n.\nastype\n(\n'float32'\n)\n# normalize to range 0-1\ntrain_norm\n=\ntrain_norm\n/\n255.0\ntest_norm\n=\ntest_norm\n/\n255.0\n# return normalized images\nreturn\ntrain_norm\n,\ntest_norm\nThis function must be called to prepare the pixel values prior to any modeling.\nDefine Model\nNext, we need a way to a neural network model.\nThe\ndefine_model()\nfunction below will define and return this model and can be filled-in or replaced for a given model configuration that we wish to evaluate later.\n# define cnn model\ndef define_model():\n\tmodel = Sequential()\n\t# ...\n\treturn model\n1\n2\n3\n4\n5\n# define cnn model\ndef\ndefine_model\n(\n)\n:\nmodel\n=\nSequential\n(\n)\n# ...\nreturn\nmodel\nEvaluate Model\nAfter the model is defined, we need to fit and evaluate it.\nFitting the model will require that the number of training epochs and batch size to be specified. We will use a generic 100 training epochs for now and a modest batch size of 64.\nIt is better to use a separate validation dataset, e.g. by splitting the train dataset into train and validation sets. We will not split the data in this case, and instead use the test dataset as a validation dataset to keep the example simple.\nThe test dataset can be used like a validation dataset and evaluated at the end of each training epoch. This will result in a trace of model evaluation scores on the train and test dataset each epoch that can be plotted later.\n# fit model\nhistory = model.fit(trainX, trainY, epochs=100, batch_size=64, validation_data=(testX, testY), verbose=0)\n1\n2\n# fit model\nhistory\n=\nmodel\n.\nfit\n(\ntrainX\n,\ntrainY\n,\nepochs\n=\n100\n,\nbatch_size\n=\n64\n,\nvalidation_data\n=\n(\ntestX\n,\ntestY\n)\n,\nverbose\n=\n0\n)\nOnce the model is fit, we can evaluate it directly on the test dataset.\n# evaluate model\n_, acc = model.evaluate(testX, testY, verbose=0)\n1\n2\n# evaluate model\n_\n,\nacc\n=\nmodel\n.\nevaluate\n(\ntestX\n,\ntestY\n,\nverbose\n=\n0\n)\nPresent Results\nOnce the model has been evaluated, we can present the results.\nThere are two key aspects to present: the diagnostics of the learning behavior of the model during training and the estimation of the model performance.\nFirst, the diagnostics involve creating a line plot showing model performance on the train and test set during training. These plots are valuable for getting an idea of whether a model is overfitting, underfitting, or has a good fit for the dataset.\nWe will create a single figure with two subplots, one for loss and one for accuracy. The blue lines will indicate model performance on the training dataset and orange lines will indicate performance on the hold out test dataset. The\nsummarize_diagnostics()\nfunction below creates and shows this plot given the collected training histories. The plot is saved to file, specifically a file with the same name as the script with a ‘\npng\n‘ extension.\n# plot diagnostic learning curves\ndef summarize_diagnostics(history):\n\t# plot loss\n\tpyplot.subplot(211)\n\tpyplot.title('Cross Entropy Loss')\n\tpyplot.plot(history.history['loss'], color='blue', label='train')\n\tpyplot.plot(history.history['val_loss'], color='orange', label='test')\n\t# plot accuracy\n\tpyplot.subplot(212)\n\tpyplot.title('Classification Accuracy')\n\tpyplot.plot(history.history['accuracy'], color='blue', label='train')\n\tpyplot.plot(history.history['val_accuracy'], color='orange', label='test')\n\t# save plot to file\n\tfilename = sys.argv[0].split('/')[-1]\n\tpyplot.savefig(filename + '_plot.png')\n\tpyplot.close()\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n14\n15\n16\n# plot diagnostic learning curves\ndef\nsummarize_diagnostics\n(\nhistory\n)\n:\n# plot loss\npyplot\n.\nsubplot\n(\n211\n)\npyplot\n.\ntitle\n(\n'Cross Entropy Loss'\n)\npyplot\n.\nplot\n(\nhistory\n.\nhistory\n[\n'loss'\n]\n,\ncolor\n=\n'blue'\n,\nlabel\n=\n'train'\n)\npyplot\n.\nplot\n(\nhistory\n.\nhistory\n[\n'val_loss'\n]\n,\ncolor\n=\n'orange'\n,\nlabel\n=\n'test'\n)\n# plot accuracy\npyplot\n.\nsubplot\n(\n212\n)\npyplot\n.\ntitle\n(\n'Classification Accuracy'\n)\npyplot\n.\nplot\n(\nhistory\n.\nhistory\n[\n'accuracy'\n]\n,\ncolor\n=\n'blue'\n,\nlabel\n=\n'train'\n)\npyplot\n.\nplot\n(\nhistory\n.\nhistory\n[\n'val_accuracy'\n]\n,\ncolor\n=\n'orange'\n,\nlabel\n=\n'test'\n)\n# save plot to file\nfilename\n=\nsys\n.\nargv\n[\n0\n]\n.\nsplit\n(\n'/'\n)\n[\n-\n1\n]\npyplot\n.\nsavefig\n(\nfilename\n+\n'_plot.png'\n)\npyplot\n.\nclose\n(\n)\nNext, we can report the final model performance on the test dataset.\nThis can be achieved by printing the classification accuracy directly.\nprint('> %.3f' % (acc * 100.0))\n1\nprint\n(\n'> %.3f'\n%\n(\nacc *\n100.0\n)\n)\nComplete Example\nWe need a function that will drive the test harness.\nThis involves calling all the define functions. The\nrun_test_harness()\nfunction below implements this and can be called to kick-off the evaluation of a given model.\n# run the test harness for evaluating a model\ndef run_test_harness():\n\t# load dataset\n\ttrainX, trainY, testX, testY = load_dataset()\n\t# prepare pixel data\n\ttrainX, testX = prep_pixels(trainX, testX)\n\t# define model\n\tmodel = define_model()\n\t# fit model\n\thistory = model.fit(trainX, trainY, epochs=100, batch_size=64, validation_data=(testX, testY), verbose=0)\n\t# evaluate model\n\t_, acc = model.evaluate(testX, testY, verbose=0)\n\tprint('> %.3f' % (acc * 100.0))\n\t# learning curves\n\tsummarize_diagnostics(history)\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n14\n15\n# run the test harness for evaluating a model\ndef\nrun_test_harness\n(\n)\n:\n# load dataset\ntrainX\n,\ntrainY\n,\ntestX\n,\ntestY\n=\nload_dataset\n(\n)\n# prepare pixel data\ntrainX\n,\ntestX\n=\nprep_pixels\n(\ntrainX\n,\ntestX\n)\n# define model\nmodel\n=\ndefine_model\n(\n)\n# fit model\nhistory\n=\nmodel\n.\nfit\n(\ntrainX\n,\ntrainY\n,\nepochs\n=\n100\n,\nbatch_size\n=\n64\n,\nvalidation_data\n=\n(\ntestX\n,\ntestY\n)\n,\nverbose\n=\n0\n)\n# evaluate model\n_\n,\nacc\n=\nmodel\n.\nevaluate\n(\ntestX\n,\ntestY\n,\nverbose\n=\n0\n)\nprint\n(\n'> %.3f'\n%\n(\nacc *\n100.0\n)\n)\n# learning curves\nsummarize_diagnostics\n(\nhistory\n)\nWe now have everything we need for the test harness.\nThe complete code example for the test harness for the CIFAR-10 dataset is listed below.\n# test harness for evaluating models on the cifar10 dataset\nimport sys\nfrom matplotlib import pyplot\nfrom keras.datasets import cifar10\nfrom keras.utils import to_categorical\nfrom keras.models import Sequential\nfrom keras.layers import Conv2D\nfrom keras.layers import MaxPooling2D\nfrom keras.layers import Dense\nfrom keras.layers import Flatten\nfrom keras.optimizers import SGD\n\n# load train and test dataset\ndef load_dataset():\n\t# load dataset\n\t(trainX, trainY), (testX, testY) = cifar10.load_data()\n\t# one hot encode target values\n\ttrainY = to_categorical(trainY)\n\ttestY = to_categorical(testY)\n\treturn trainX, trainY, testX, testY\n\n# scale pixels\ndef prep_pixels(train, test):\n\t# convert from integers to floats\n\ttrain_norm = train.astype('float32')\n\ttest_norm = test.astype('float32')\n\t# normalize to range 0-1\n\ttrain_norm = train_norm / 255.0\n\ttest_norm = test_norm / 255.0\n\t# return normalized images\n\treturn train_norm, test_norm\n\n# define cnn model\ndef define_model():\n\tmodel = Sequential()\n\t# ...\n\treturn model\n\n# plot diagnostic learning curves\ndef summarize_diagnostics(history):\n\t# plot loss\n\tpyplot.subplot(211)\n\tpyplot.title('Cross Entropy Loss')\n\tpyplot.plot(history.history['loss'], color='blue', label='train')\n\tpyplot.plot(history.history['val_loss'], color='orange', label='test')\n\t# plot accuracy\n\tpyplot.subplot(212)\n\tpyplot.title('Classification Accuracy')\n\tpyplot.plot(history.history['accuracy'], color='blue', label='train')\n\tpyplot.plot(history.history['val_accuracy'], color='orange', label='test')\n\t# save plot to file\n\tfilename = sys.argv[0].split('/')[-1]\n\tpyplot.savefig(filename + '_plot.png')\n\tpyplot.close()\n\n# run the test harness for evaluating a model\ndef run_test_harness():\n\t# load dataset\n\ttrainX, trainY, testX, testY = load_dataset()\n\t# prepare pixel data\n\ttrainX, testX = prep_pixels(trainX, testX)\n\t# define model\n\tmodel = define_model()\n\t# fit model\n\thistory = model.fit(trainX, trainY, epochs=100, batch_size=64, validation_data=(testX, testY), verbose=0)\n\t# evaluate model\n\t_, acc = model.evaluate(testX, testY, verbose=0)\n\tprint('> %.3f' % (acc * 100.0))\n\t# learning curves\n\tsummarize_diagnostics(history)\n\n# entry point, run the test harness\nrun_test_harness()\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n14\n15\n16\n17\n18\n19\n20\n21\n22\n23\n24\n25\n26\n27\n28\n29\n30\n31\n32\n33\n34\n35\n36\n37\n38\n39\n40\n41\n42\n43\n44\n45\n46\n47\n48\n49\n50\n51\n52\n53\n54\n55\n56\n57\n58\n59\n60\n61\n62\n63\n64\n65\n66\n67\n68\n69\n70\n71\n72\n73\n# test harness for evaluating models on the cifar10 dataset\nimport\nsys\nfrom\nmatplotlib\nimport\npyplot\nfrom\nkeras\n.\ndatasets\nimport\ncifar10\nfrom\nkeras\n.\nutils\nimport\nto_categorical\nfrom\nkeras\n.\nmodels\nimport\nSequential\nfrom\nkeras\n.\nlayers\nimport\nConv2D\nfrom\nkeras\n.\nlayers\nimport\nMaxPooling2D\nfrom\nkeras\n.\nlayers\nimport\nDense\nfrom\nkeras\n.\nlayers\nimport\nFlatten\nfrom\nkeras\n.\noptimizers\nimport\nSGD\n# load train and test dataset\ndef\nload_dataset\n(\n)\n:\n# load dataset\n(\ntrainX\n,\ntrainY\n)\n,\n(\ntestX\n,\ntestY\n)\n=\ncifar10\n.\nload_data\n(\n)\n# one hot encode target values\ntrainY\n=\nto_categorical\n(\ntrainY\n)\ntestY\n=\nto_categorical\n(\ntestY\n)\nreturn\ntrainX\n,\ntrainY\n,\ntestX\n,\ntestY\n# scale pixels\ndef\nprep_pixels\n(\ntrain\n,\ntest\n)\n:\n# convert from integers to floats\ntrain_norm\n=\ntrain\n.\nastype\n(\n'float32'\n)\ntest_norm\n=\ntest\n.\nastype\n(\n'float32'\n)\n# normalize to range 0-1\ntrain_norm\n=\ntrain_norm\n/\n255.0\ntest_norm\n=\ntest_norm\n/\n255.0\n# return normalized images\nreturn\ntrain_norm\n,\ntest\n_\nnorm\n# define cnn model\ndef\ndefine_model\n(\n)\n:\nmodel\n=\nSequential\n(\n)\n# ...\nreturn\nmodel\n# plot diagnostic learning curves\ndef\nsummarize_diagnostics\n(\nhistory\n)\n:\n# plot loss\npyplot\n.\nsubplot\n(\n211\n)\npyplot\n.\ntitle\n(\n'Cross Entropy Loss'\n)\npyplot\n.\nplot\n(\nhistory\n.\nhistory\n[\n'loss'\n]\n,\ncolor\n=\n'blue'\n,\nlabel\n=\n'train'\n)\npyplot\n.\nplot\n(\nhistory\n.\nhistory\n[\n'val_loss'\n]\n,\ncolor\n=\n'orange'\n,\nlabel\n=\n'test'\n)\n# plot accuracy\npyplot\n.\nsubplot\n(\n212\n)\npyplot\n.\ntitle\n(\n'Classification Accuracy'\n)\npyplot\n.\nplot\n(\nhistory\n.\nhistory\n[\n'accuracy'\n]\n,\ncolor\n=\n'blue'\n,\nlabel\n=\n'train'\n)\npyplot\n.\nplot\n(\nhistory\n.\nhistory\n[\n'val_accuracy'\n]\n,\ncolor\n=\n'orange'\n,\nlabel\n=\n'test'\n)\n# save plot to file\nfilename\n=\nsys\n.\nargv\n[\n0\n]\n.\nsplit\n(\n'/'\n)\n[\n-\n1\n]\npyplot\n.\nsavefig\n(\nfilename\n+\n'_plot.png'\n)\npyplot\n.\nclose\n(\n)\n# run the test harness for evaluating a model\ndef\nrun_test_harness\n(\n)\n:\n# load dataset\ntrainX\n,\ntrainY\n,\ntestX\n,\ntestY\n=\nload_dataset\n(\n)\n# prepare pixel data\ntrainX\n,\ntestX\n=\nprep_pixels\n(\ntrainX\n,\ntestX\n)\n# define model\nmodel\n=\ndefine_model\n(\n)\n# fit model\nhistory\n=\nmodel\n.\nfit\n(\ntrainX\n,\ntrainY\n,\nepochs\n=\n100\n,\nbatch_size\n=\n64\n,\nvalidation_data\n=\n(\ntestX\n,\ntestY\n)\n,\nverbose\n=\n0\n)\n# evaluate model\n_\n,\nacc\n=\nmodel\n.\nevaluate\n(\ntestX\n,\ntestY\n,\nverbose\n=\n0\n)\nprint\n(\n'> %.3f'\n%\n(\nacc *\n100.0\n)\n)\n# learning curves\nsummarize_diagnostics\n(\nhistory\n)\n# entry point, run the test harness\nrun_test_harness\n(\n)\nThis test harness can evaluate any CNN models we may wish to evaluate on the CIFAR-10 dataset and can run on the CPU or GPU.\nNote\n: as is, no model is defined, so this complete example cannot be run.\nNext, let’s look at how we can define and evaluate a baseline model.\nHow to Develop a Baseline Model\nWe can now investigate a baseline model for the CIFAR-10 dataset.\nA baseline model will establish a minimum model performance to which all of our other models can be compared, as well as a model architecture that we can use as the basis of study and improvement.\nA good starting point is the general architectural principles of the VGG models. These are a good starting point because they achieved top performance in the ILSVRC 2014 competition and because the modular structure of the architecture is easy to understand and implement. For more details on the VGG model, see the 2015 paper “\nVery Deep Convolutional Networks for Large-Scale Image Recognition\n.”\nThe architecture involves stacking convolutional layers with small 3×3 filters followed by a max pooling layer. Together, these layers form a block, and these blocks can be repeated where the number of filters in each block is increased with the depth of the network such as 32, 64, 128, 256 for the first four blocks of the model. Padding is used on the convolutional layers to ensure the height and width of the output feature maps matches the inputs.\nWe can explore this architecture on the CIFAR-10 problem and compare a model with this architecture with 1, 2, and 3 blocks.\nEach layer will use the\nReLU activation function\nand the He weight initialization, which are generally best practices. For example, a 3-block VGG-style architecture can be defined in Keras as follows:\n# example of a 3-block vgg style architecture\nmodel = Sequential()\nmodel.add(Conv2D(32, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same', input_shape=(32, 32, 3)))\nmodel.add(Conv2D(32, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))\nmodel.add(MaxPooling2D((2, 2)))\nmodel.add(Conv2D(64, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))\nmodel.add(Conv2D(64, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))\nmodel.add(MaxPooling2D((2, 2)))\nmodel.add(Conv2D(128, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))\nmodel.add(Conv2D(128, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))\nmodel.add(MaxPooling2D((2, 2)))\n...\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n# example of a 3-block vgg style architecture\nmodel\n=\nSequential\n(\n)\nmodel\n.\nadd\n(\nConv2D\n(\n32\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n,\ninput_shape\n=\n(\n32\n,\n32\n,\n3\n)\n)\n)\nmodel\n.\nadd\n(\nConv2D\n(\n32\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n)\n)\nmodel\n.\nadd\n(\nMaxPooling2D\n(\n(\n2\n,\n2\n)\n)\n)\nmodel\n.\nadd\n(\nConv2D\n(\n64\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n)\n)\nmodel\n.\nadd\n(\nConv2D\n(\n64\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n)\n)\nmodel\n.\nadd\n(\nMaxPooling2D\n(\n(\n2\n,\n2\n)\n)\n)\nmodel\n.\nadd\n(\nConv2D\n(\n128\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n)\n)\nmodel\n.\nadd\n(\nConv2D\n(\n128\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n)\n)\nmodel\n.\nadd\n(\nMaxPooling2D\n(\n(\n2\n,\n2\n)\n)\n)\n.\n.\n.\nThis defines the feature detector part of the model. This must be coupled with a classifier part of the model that interprets the features and makes a prediction as to which class a given photo belongs.\nThis can be fixed for each model that we investigate. First, the feature maps output from the feature extraction part of the model must be flattened. We can then interpret them with one or more fully connected layers, and then output a prediction. The output layer must have 10 nodes for the 10 classes and use the softmax activation function.\n# example output part of the model\nmodel.add(Flatten())\nmodel.add(Dense(128, activation='relu', kernel_initializer='he_uniform'))\nmodel.add(Dense(10, activation='softmax'))\n...\n1\n2\n3\n4\n5\n# example output part of the model\nmodel\n.\nadd\n(\nFlatten\n(\n)\n)\nmodel\n.\nadd\n(\nDense\n(\n128\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n)\n)\nmodel\n.\nadd\n(\nDense\n(\n10\n,\nactivation\n=\n'softmax'\n)\n)\n.\n.\n.\nThe model will be optimized using stochastic gradient descent.\nWe will use a modest learning rate of 0.001 and a large momentum of 0.9, both of which are good general starting points. The model will optimize the\ncategorical cross entropy loss function\nrequired for multi-class classification and will monitor classification accuracy.\n# compile model\nopt = SGD(lr=0.001, momentum=0.9)\nmodel.compile(optimizer=opt, loss='categorical_crossentropy', metrics=['accuracy'])\n1\n2\n3\n# compile model\nopt\n=\nSGD\n(\nlr\n=\n0.001\n,\nmomentum\n=\n0.9\n)\nmodel\n.\ncompile\n(\noptimizer\n=\nopt\n,\nloss\n=\n'categorical_crossentropy'\n,\nmetrics\n=\n[\n'accuracy'\n]\n)\nWe now have enough elements to define our VGG-style baseline models. We can define three different model architectures with 1, 2, and 3 VGG modules which requires that we define 3 separate versions of the\ndefine_model()\nfunction, provided below.\nTo test each model, a new script must be created (e.g.\nmodel_baseline1.py\n,\nmodel_baseline2.py\n, …) using the test harness defined in the previous section, and with the new version of the\ndefine_model()\nfunction defined below.\nLet’s take a look at each\ndefine_model()\nfunction and the evaluation of the resulting test harness in turn.\nBaseline: 1 VGG Block\nThe\ndefine_model()\nfunction for one VGG block is listed below.\n# define cnn model\ndef define_model():\n\tmodel = Sequential()\n\tmodel.add(Conv2D(32, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same', input_shape=(32, 32, 3)))\n\tmodel.add(Conv2D(32, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))\n\tmodel.add(MaxPooling2D((2, 2)))\n\tmodel.add(Flatten())\n\tmodel.add(Dense(128, activation='relu', kernel_initializer='he_uniform'))\n\tmodel.add(Dense(10, activation='softmax'))\n\t# compile model\n\topt = SGD(lr=0.001, momentum=0.9)\n\tmodel.compile(optimizer=opt, loss='categorical_crossentropy', metrics=['accuracy'])\n\treturn model\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n# define cnn model\ndef\ndefine_model\n(\n)\n:\nmodel\n=\nSequential\n(\n)\nmodel\n.\nadd\n(\nConv2D\n(\n32\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n,\ninput_shape\n=\n(\n32\n,\n32\n,\n3\n)\n)\n)\nmodel\n.\nadd\n(\nConv2D\n(\n32\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n)\n)\nmodel\n.\nadd\n(\nMaxPooling2D\n(\n(\n2\n,\n2\n)\n)\n)\nmodel\n.\nadd\n(\nFlatten\n(\n)\n)\nmodel\n.\nadd\n(\nDense\n(\n128\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n)\n)\nmodel\n.\nadd\n(\nDense\n(\n10\n,\nactivation\n=\n'softmax'\n)\n)\n# compile model\nopt\n=\nSGD\n(\nlr\n=\n0.001\n,\nmomentum\n=\n0.9\n)\nmodel\n.\ncompile\n(\noptimizer\n=\nopt\n,\nloss\n=\n'categorical_crossentropy'\n,\nmetrics\n=\n[\n'accuracy'\n]\n)\nreturn\nmodel\nRunning the model in the test harness first prints the classification accuracy on the test dataset.\nNote\n: Your\nresults may vary\ngiven the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.\nIn this case, we can see that the model achieved a classification accuracy of just less than 70%.\n> 67.070\n1\n> 67.070\nA figure is created and saved to file showing the learning curves of the model during training on the train and test dataset, both with regards to the loss and accuracy.\nIn this case, we can see that the model rapidly overfits the test dataset. This is clear if we look at the plot of loss (top plot), we can see that the model’s performance on the training dataset (blue) continues to improve whereas the performance on the test dataset (orange) improves, then starts to get worse at around 15 epochs.\nLine Plots of Learning Curves for VGG 1 Baseline on the CIFAR-10 Dataset\nBaseline: 2 VGG Blocks\nThe\ndefine_model()\nfunction for two VGG blocks is listed below.\n# define cnn model\ndef define_model():\n\tmodel = Sequential()\n\tmodel.add(Conv2D(32, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same', input_shape=(32, 32, 3)))\n\tmodel.add(Conv2D(32, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))\n\tmodel.add(MaxPooling2D((2, 2)))\n\tmodel.add(Conv2D(64, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))\n\tmodel.add(Conv2D(64, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))\n\tmodel.add(MaxPooling2D((2, 2)))\n\tmodel.add(Flatten())\n\tmodel.add(Dense(128, activation='relu', kernel_initializer='he_uniform'))\n\tmodel.add(Dense(10, activation='softmax'))\n\t# compile model\n\topt = SGD(lr=0.001, momentum=0.9)\n\tmodel.compile(optimizer=opt, loss='categorical_crossentropy', metrics=['accuracy'])\n\treturn model\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n14\n15\n16\n# define cnn model\ndef\ndefine_model\n(\n)\n:\nmodel\n=\nSequential\n(\n)\nmodel\n.\nadd\n(\nConv2D\n(\n32\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n,\ninput_shape\n=\n(\n32\n,\n32\n,\n3\n)\n)\n)\nmodel\n.\nadd\n(\nConv2D\n(\n32\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n)\n)\nmodel\n.\nadd\n(\nMaxPooling2D\n(\n(\n2\n,\n2\n)\n)\n)\nmodel\n.\nadd\n(\nConv2D\n(\n64\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n)\n)\nmodel\n.\nadd\n(\nConv2D\n(\n64\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n)\n)\nmodel\n.\nadd\n(\nMaxPooling2D\n(\n(\n2\n,\n2\n)\n)\n)\nmodel\n.\nadd\n(\nFlatten\n(\n)\n)\nmodel\n.\nadd\n(\nDense\n(\n128\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n)\n)\nmodel\n.\nadd\n(\nDense\n(\n10\n,\nactivation\n=\n'softmax'\n)\n)\n# compile model\nopt\n=\nSGD\n(\nlr\n=\n0.001\n,\nmomentum\n=\n0.9\n)\nmodel\n.\ncompile\n(\noptimizer\n=\nopt\n,\nloss\n=\n'categorical_crossentropy'\n,\nmetrics\n=\n[\n'accuracy'\n]\n)\nreturn\nmodel\nRunning the model in the test harness first prints the classification accuracy on the test dataset.\nNote\n: Your\nresults may vary\ngiven the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.\nIn this case, we can see that the model with two blocks performs better than the model with a single block: a good sign.\n> 71.080\n1\n> 71.080\nA figure showing learning curves is created and saved to file. In this case, we continue to see strong overfitting.\nLine Plots of Learning Curves for VGG 2 Baseline on the CIFAR-10 Dataset\nBaseline: 3 VGG Blocks\nThe\ndefine_model()\nfunction for three VGG blocks is listed below.\n# define cnn model\ndef define_model():\n\tmodel = Sequential()\n\tmodel.add(Conv2D(32, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same', input_shape=(32, 32, 3)))\n\tmodel.add(Conv2D(32, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))\n\tmodel.add(MaxPooling2D((2, 2)))\n\tmodel.add(Conv2D(64, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))\n\tmodel.add(Conv2D(64, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))\n\tmodel.add(MaxPooling2D((2, 2)))\n\tmodel.add(Conv2D(128, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))\n\tmodel.add(Conv2D(128, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))\n\tmodel.add(MaxPooling2D((2, 2)))\n\tmodel.add(Flatten())\n\tmodel.add(Dense(128, activation='relu', kernel_initializer='he_uniform'))\n\tmodel.add(Dense(10, activation='softmax'))\n\t# compile model\n\topt = SGD(lr=0.001, momentum=0.9)\n\tmodel.compile(optimizer=opt, loss='categorical_crossentropy', metrics=['accuracy'])\n\treturn model\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n14\n15\n16\n17\n18\n19\n# define cnn model\ndef\ndefine_model\n(\n)\n:\nmodel\n=\nSequential\n(\n)\nmodel\n.\nadd\n(\nConv2D\n(\n32\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n,\ninput_shape\n=\n(\n32\n,\n32\n,\n3\n)\n)\n)\nmodel\n.\nadd\n(\nConv2D\n(\n32\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n)\n)\nmodel\n.\nadd\n(\nMaxPooling2D\n(\n(\n2\n,\n2\n)\n)\n)\nmodel\n.\nadd\n(\nConv2D\n(\n64\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n)\n)\nmodel\n.\nadd\n(\nConv2D\n(\n64\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n)\n)\nmodel\n.\nadd\n(\nMaxPooling2D\n(\n(\n2\n,\n2\n)\n)\n)\nmodel\n.\nadd\n(\nConv2D\n(\n128\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n)\n)\nmodel\n.\nadd\n(\nConv2D\n(\n128\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n)\n)\nmodel\n.\nadd\n(\nMaxPooling2D\n(\n(\n2\n,\n2\n)\n)\n)\nmodel\n.\nadd\n(\nFlatten\n(\n)\n)\nmodel\n.\nadd\n(\nDense\n(\n128\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n)\n)\nmodel\n.\nadd\n(\nDense\n(\n10\n,\nactivation\n=\n'softmax'\n)\n)\n# compile model\nopt\n=\nSGD\n(\nlr\n=\n0.001\n,\nmomentum\n=\n0.9\n)\nmodel\n.\ncompile\n(\noptimizer\n=\nopt\n,\nloss\n=\n'categorical_crossentropy'\n,\nmetrics\n=\n[\n'accuracy'\n]\n)\nreturn\nmodel\nRunning the model in the test harness first prints the classification accuracy on the test dataset.\nNote\n: Your\nresults may vary\ngiven the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.\nIn this case, yet another modest increase in performance is seen as the depth of the model was increased.\n> 73.500\n1\n> 73.500\nReviewing the figures showing the learning curves, again we see dramatic overfitting within the first 20 training epochs.\nLine Plots of Learning Curves for VGG 3 Baseline on the CIFAR-10 Dataset\nDiscussion\nWe have explored three different models with a VGG-based architecture.\nThe results can be summarized below, although we must assume some variance in these results given the stochastic nature of the algorithm:\nVGG 1\n: 67.070%\nVGG 2\n: 71.080%\nVGG 3\n: 73.500%\nIn all cases, the model was able to learn the training dataset, showing an improvement on the training dataset that at least continued to 40 epochs, and perhaps more. This is a good sign, as it shows that the problem is learnable and that all three models have sufficient capacity to learn the problem.\nThe results of the model on the test dataset showed an improvement in classification accuracy with each increase in the depth of the model. It is possible that this trend would continue if models with four and five layers were evaluated, and this might make an interesting extension. Nevertheless, all three models showed the same pattern of dramatic overfitting at around 15-to-20 epochs.\nThese results suggest that the model with three VGG blocks is a good starting point or baseline model for our investigation.\nThe results also suggest that the model is in need of regularization to address the rapid overfitting of the test dataset. More generally, the results suggest that it may be useful to investigate techniques that slow down the convergence (rate of learning) of the model. This may include techniques such as data augmentation as well as learning rate schedules, changes to the batch size, and perhaps more.\nIn the next section, we will investigate some of these ideas for improving model performance.\nHow to Develop an Improved Model\nNow that we have established a baseline model, the VGG architecture with three blocks, we can investigate modifications to the model and the training algorithm that seek to improve performance.\nWe will look at two main areas first to address the severe overfitting observed, namely regularization and data augmentation.\nRegularization Techniques\nThere are many regularization techniques we could try, although the nature of the overfitting observed suggests that perhaps early stopping would not be appropriate and that techniques that slow down the rate of convergence might be useful.\nWe will look into the effect of both dropout and weight regularization or weight decay.\nDropout Regularization\nDropout is a simple technique that will randomly drop nodes out of the network. It has a regularizing effect as the remaining nodes must adapt to pick-up the slack of the removed nodes.\nFor more on dropout, see the post:\nA Gentle Introduction to Dropout for Regularizing Deep Neural Networks\nDropout can be added to the model by adding new Dropout layers, where the amount of nodes removed is specified as a parameter. There are many patterns for adding Dropout to a model, in terms of where in the model to add the layers and how much dropout to use.\nIn this case, we will add Dropout layers after each max pooling layer and after the fully connected layer, and use a fixed dropout rate of 20% (e.g. retain 80% of the nodes).\nThe updated VGG 3 baseline model with dropout is listed below.\n# define cnn model\ndef define_model():\n\tmodel = Sequential()\n\tmodel.add(Conv2D(32, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same', input_shape=(32, 32, 3)))\n\tmodel.add(Conv2D(32, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))\n\tmodel.add(MaxPooling2D((2, 2)))\n\tmodel.add(Dropout(0.2))\n\tmodel.add(Conv2D(64, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))\n\tmodel.add(Conv2D(64, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))\n\tmodel.add(MaxPooling2D((2, 2)))\n\tmodel.add(Dropout(0.2))\n\tmodel.add(Conv2D(128, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))\n\tmodel.add(Conv2D(128, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))\n\tmodel.add(MaxPooling2D((2, 2)))\n\tmodel.add(Dropout(0.2))\n\tmodel.add(Flatten())\n\tmodel.add(Dense(128, activation='relu', kernel_initializer='he_uniform'))\n\tmodel.add(Dropout(0.2))\n\tmodel.add(Dense(10, activation='softmax'))\n\t# compile model\n\topt = SGD(lr=0.001, momentum=0.9)\n\tmodel.compile(optimizer=opt, loss='categorical_crossentropy', metrics=['accuracy'])\n\treturn model\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n14\n15\n16\n17\n18\n19\n20\n21\n22\n23\n# define cnn model\ndef\ndefine_model\n(\n)\n:\nmodel\n=\nSequential\n(\n)\nmodel\n.\nadd\n(\nConv2D\n(\n32\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n,\ninput_shape\n=\n(\n32\n,\n32\n,\n3\n)\n)\n)\nmodel\n.\nadd\n(\nConv2D\n(\n32\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n)\n)\nmodel\n.\nadd\n(\nMaxPooling2D\n(\n(\n2\n,\n2\n)\n)\n)\nmodel\n.\nadd\n(\nDropout\n(\n0.2\n)\n)\nmodel\n.\nadd\n(\nConv2D\n(\n64\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n)\n)\nmodel\n.\nadd\n(\nConv2D\n(\n64\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n)\n)\nmodel\n.\nadd\n(\nMaxPooling2D\n(\n(\n2\n,\n2\n)\n)\n)\nmodel\n.\nadd\n(\nDropout\n(\n0.2\n)\n)\nmodel\n.\nadd\n(\nConv2D\n(\n128\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n)\n)\nmodel\n.\nadd\n(\nConv2D\n(\n128\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n)\n)\nmodel\n.\nadd\n(\nMaxPooling2D\n(\n(\n2\n,\n2\n)\n)\n)\nmodel\n.\nadd\n(\nDropout\n(\n0.2\n)\n)\nmodel\n.\nadd\n(\nFlatten\n(\n)\n)\nmodel\n.\nadd\n(\nDense\n(\n128\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n)\n)\nmodel\n.\nadd\n(\nDropout\n(\n0.2\n)\n)\nmodel\n.\nadd\n(\nDense\n(\n10\n,\nactivation\n=\n'softmax'\n)\n)\n# compile model\nopt\n=\nSGD\n(\nlr\n=\n0.001\n,\nmomentum\n=\n0.9\n)\nmodel\n.\ncompile\n(\noptimizer\n=\nopt\n,\nloss\n=\n'categorical_crossentropy'\n,\nmetrics\n=\n[\n'accuracy'\n]\n)\nreturn\nmodel\nThe full code listing is provided below for completeness.\n# baseline model with dropout on the cifar10 dataset\nimport sys\nfrom matplotlib import pyplot\nfrom keras.datasets import cifar10\nfrom keras.utils import to_categorical\nfrom keras.models import Sequential\nfrom keras.layers import Conv2D\nfrom keras.layers import MaxPooling2D\nfrom keras.layers import Dense\nfrom keras.layers import Flatten\nfrom keras.layers import Dropout\nfrom keras.optimizers import SGD\n\n# load train and test dataset\ndef load_dataset():\n\t# load dataset\n\t(trainX, trainY), (testX, testY) = cifar10.load_data()\n\t# one hot encode target values\n\ttrainY = to_categorical(trainY)\n\ttestY = to_categorical(testY)\n\treturn trainX, trainY, testX, testY\n\n# scale pixels\ndef prep_pixels(train, test):\n\t# convert from integers to floats\n\ttrain_norm = train.astype('float32')\n\ttest_norm = test.astype('float32')\n\t# normalize to range 0-1\n\ttrain_norm = train_norm / 255.0\n\ttest_norm = test_norm / 255.0\n\t# return normalized images\n\treturn train_norm, test_norm\n\n# define cnn model\ndef define_model():\n\tmodel = Sequential()\n\tmodel.add(Conv2D(32, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same', input_shape=(32, 32, 3)))\n\tmodel.add(Conv2D(32, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))\n\tmodel.add(MaxPooling2D((2, 2)))\n\tmodel.add(Dropout(0.2))\n\tmodel.add(Conv2D(64, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))\n\tmodel.add(Conv2D(64, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))\n\tmodel.add(MaxPooling2D((2, 2)))\n\tmodel.add(Dropout(0.2))\n\tmodel.add(Conv2D(128, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))\n\tmodel.add(Conv2D(128, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))\n\tmodel.add(MaxPooling2D((2, 2)))\n\tmodel.add(Dropout(0.2))\n\tmodel.add(Flatten())\n\tmodel.add(Dense(128, activation='relu', kernel_initializer='he_uniform'))\n\tmodel.add(Dropout(0.2))\n\tmodel.add(Dense(10, activation='softmax'))\n\t# compile model\n\topt = SGD(lr=0.001, momentum=0.9)\n\tmodel.compile(optimizer=opt, loss='categorical_crossentropy', metrics=['accuracy'])\n\treturn model\n\n# plot diagnostic learning curves\ndef summarize_diagnostics(history):\n\t# plot loss\n\tpyplot.subplot(211)\n\tpyplot.title('Cross Entropy Loss')\n\tpyplot.plot(history.history['loss'], color='blue', label='train')\n\tpyplot.plot(history.history['val_loss'], color='orange', label='test')\n\t# plot accuracy\n\tpyplot.subplot(212)\n\tpyplot.title('Classification Accuracy')\n\tpyplot.plot(history.history['accuracy'], color='blue', label='train')\n\tpyplot.plot(history.history['val_accuracy'], color='orange', label='test')\n\t# save plot to file\n\tfilename = sys.argv[0].split('/')[-1]\n\tpyplot.savefig(filename + '_plot.png')\n\tpyplot.close()\n\n# run the test harness for evaluating a model\ndef run_test_harness():\n\t# load dataset\n\ttrainX, trainY, testX, testY = load_dataset()\n\t# prepare pixel data\n\ttrainX, testX = prep_pixels(trainX, testX)\n\t# define model\n\tmodel = define_model()\n\t# fit model\n\thistory = model.fit(trainX, trainY, epochs=100, batch_size=64, validation_data=(testX, testY), verbose=0)\n\t# evaluate model\n\t_, acc = model.evaluate(testX, testY, verbose=0)\n\tprint('> %.3f' % (acc * 100.0))\n\t# learning curves\n\tsummarize_diagnostics(history)\n\n# entry point, run the test harness\nrun_test_harness()\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n14\n15\n16\n17\n18\n19\n20\n21\n22\n23\n24\n25\n26\n27\n28\n29\n30\n31\n32\n33\n34\n35\n36\n37\n38\n39\n40\n41\n42\n43\n44\n45\n46\n47\n48\n49\n50\n51\n52\n53\n54\n55\n56\n57\n58\n59\n60\n61\n62\n63\n64\n65\n66\n67\n68\n69\n70\n71\n72\n73\n74\n75\n76\n77\n78\n79\n80\n81\n82\n83\n84\n85\n86\n87\n88\n89\n90\n91\n92\n# baseline model with dropout on the cifar10 dataset\nimport\nsys\nfrom\nmatplotlib\nimport\npyplot\nfrom\nkeras\n.\ndatasets\nimport\ncifar10\nfrom\nkeras\n.\nutils\nimport\nto_categorical\nfrom\nkeras\n.\nmodels\nimport\nSequential\nfrom\nkeras\n.\nlayers\nimport\nConv2D\nfrom\nkeras\n.\nlayers\nimport\nMaxPooling2D\nfrom\nkeras\n.\nlayers\nimport\nDense\nfrom\nkeras\n.\nlayers\nimport\nFlatten\nfrom\nkeras\n.\nlayers\nimport\nDropout\nfrom\nkeras\n.\noptimizers\nimport\nSGD\n# load train and test dataset\ndef\nload_dataset\n(\n)\n:\n# load dataset\n(\ntrainX\n,\ntrainY\n)\n,\n(\ntestX\n,\ntestY\n)\n=\ncifar10\n.\nload_data\n(\n)\n# one hot encode target values\ntrainY\n=\nto_categorical\n(\ntrainY\n)\ntestY\n=\nto_categorical\n(\ntestY\n)\nreturn\ntrainX\n,\ntrainY\n,\ntestX\n,\ntestY\n# scale pixels\ndef\nprep_pixels\n(\ntrain\n,\ntest\n)\n:\n# convert from integers to floats\ntrain_norm\n=\ntrain\n.\nastype\n(\n'float32'\n)\ntest_norm\n=\ntest\n.\nastype\n(\n'float32'\n)\n# normalize to range 0-1\ntrain_norm\n=\ntrain_norm\n/\n255.0\ntest_norm\n=\ntest_norm\n/\n255.0\n# return normalized images\nreturn\ntrain_norm\n,\ntest\n_\nnorm\n# define cnn model\ndef\ndefine_model\n(\n)\n:\nmodel\n=\nSequential\n(\n)\nmodel\n.\nadd\n(\nConv2D\n(\n32\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n,\ninput_shape\n=\n(\n32\n,\n32\n,\n3\n)\n)\n)\nmodel\n.\nadd\n(\nConv2D\n(\n32\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n)\n)\nmodel\n.\nadd\n(\nMaxPooling2D\n(\n(\n2\n,\n2\n)\n)\n)\nmodel\n.\nadd\n(\nDropout\n(\n0.2\n)\n)\nmodel\n.\nadd\n(\nConv2D\n(\n64\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n)\n)\nmodel\n.\nadd\n(\nConv2D\n(\n64\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n)\n)\nmodel\n.\nadd\n(\nMaxPooling2D\n(\n(\n2\n,\n2\n)\n)\n)\nmodel\n.\nadd\n(\nDropout\n(\n0.2\n)\n)\nmodel\n.\nadd\n(\nConv2D\n(\n128\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n)\n)\nmodel\n.\nadd\n(\nConv2D\n(\n128\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n)\n)\nmodel\n.\nadd\n(\nMaxPooling2D\n(\n(\n2\n,\n2\n)\n)\n)\nmodel\n.\nadd\n(\nDropout\n(\n0.2\n)\n)\nmodel\n.\nadd\n(\nFlatten\n(\n)\n)\nmodel\n.\nadd\n(\nDense\n(\n128\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n)\n)\nmodel\n.\nadd\n(\nDropout\n(\n0.2\n)\n)\nmodel\n.\nadd\n(\nDense\n(\n10\n,\nactivation\n=\n'softmax'\n)\n)\n# compile model\nopt\n=\nSGD\n(\nlr\n=\n0.001\n,\nmomentum\n=\n0.9\n)\nmodel\n.\ncompile\n(\noptimizer\n=\nopt\n,\nloss\n=\n'categorical_crossentropy'\n,\nmetrics\n=\n[\n'accuracy'\n]\n)\nreturn\nmodel\n# plot diagnostic learning curves\ndef\nsummarize_diagnostics\n(\nhistory\n)\n:\n# plot loss\npyplot\n.\nsubplot\n(\n211\n)\npyplot\n.\ntitle\n(\n'Cross Entropy Loss'\n)\npyplot\n.\nplot\n(\nhistory\n.\nhistory\n[\n'loss'\n]\n,\ncolor\n=\n'blue'\n,\nlabel\n=\n'train'\n)\npyplot\n.\nplot\n(\nhistory\n.\nhistory\n[\n'val_loss'\n]\n,\ncolor\n=\n'orange'\n,\nlabel\n=\n'test'\n)\n# plot accuracy\npyplot\n.\nsubplot\n(\n212\n)\npyplot\n.\ntitle\n(\n'Classification Accuracy'\n)\npyplot\n.\nplot\n(\nhistory\n.\nhistory\n[\n'accuracy'\n]\n,\ncolor\n=\n'blue'\n,\nlabel\n=\n'train'\n)\npyplot\n.\nplot\n(\nhistory\n.\nhistory\n[\n'val_accuracy'\n]\n,\ncolor\n=\n'orange'\n,\nlabel\n=\n'test'\n)\n# save plot to file\nfilename\n=\nsys\n.\nargv\n[\n0\n]\n.\nsplit\n(\n'/'\n)\n[\n-\n1\n]\npyplot\n.\nsavefig\n(\nfilename\n+\n'_plot.png'\n)\npyplot\n.\nclose\n(\n)\n# run the test harness for evaluating a model\ndef\nrun_test_harness\n(\n)\n:\n# load dataset\ntrainX\n,\ntrainY\n,\ntestX\n,\ntestY\n=\nload_dataset\n(\n)\n# prepare pixel data\ntrainX\n,\ntestX\n=\nprep_pixels\n(\ntrainX\n,\ntestX\n)\n# define model\nmodel\n=\ndefine_model\n(\n)\n# fit model\nhistory\n=\nmodel\n.\nfit\n(\ntrainX\n,\ntrainY\n,\nepochs\n=\n100\n,\nbatch_size\n=\n64\n,\nvalidation_data\n=\n(\ntestX\n,\ntestY\n)\n,\nverbose\n=\n0\n)\n# evaluate model\n_\n,\nacc\n=\nmodel\n.\nevaluate\n(\ntestX\n,\ntestY\n,\nverbose\n=\n0\n)\nprint\n(\n'> %.3f'\n%\n(\nacc *\n100.0\n)\n)\n# learning curves\nsummarize_diagnostics\n(\nhistory\n)\n# entry point, run the test harness\nrun_test_harness\n(\n)\nRunning the model in the test harness prints the classification accuracy on the test dataset.\nNote\n: Your\nresults may vary\ngiven the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.\nIn this case, we can see a jump in classification accuracy by about 10% from about 73% without dropout to about 83% with dropout.\n> 83.450\n1\n> 83.450\nReviewing the learning curve for the model, we can see that overfitting has been addressed. The model converges well for about 40 or 50 epochs, at which point there is no further improvement on the test dataset.\nThis is a great result. We could elaborate upon this model and add early stopping with a patience of about 10 epochs to save a well-performing model on the test set during training at around the point that no further improvements are observed.\nWe could also try exploring a learning rate schedule that drops the learning rate after improvements on the test set stall.\nDropout has performed well, and we do not know that the chosen rate of 20% is the best. We could explore other dropout rates, as well as differing positioning of the dropout layers in the model architecture.\nLine Plots of Learning Curves for Baseline Model With Dropout on the CIFAR-10 Dataset\nWeight Decay\nWeight regularization or weight decay involves updating the loss function to penalize the model in proportion to the size of the model weights.\nThis has a regularizing effect, as larger weights result in a more complex and less stable model, whereas smaller weights are often more stable and more general.\nTo learn more about weight regularization, see the post:\nUse Weight Regularization to Reduce Overfitting of Deep Learning Models\nWe can add weight regularization to the convolutional layers and the fully connected layers by defining the “\nkernel_regularizer\n” argument and specifying the type of regularization. In this case, we will use\nL2\nweight regularization, the most common type used for neural networks and a sensible default weighting of 0.001.\nThe updated baseline model with weight decay is listed below.\n# define cnn model\ndef define_model():\n\tmodel = Sequential()\n\tmodel.add(Conv2D(32, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same', kernel_regularizer=l2(0.001), input_shape=(32, 32, 3)))\n\tmodel.add(Conv2D(32, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same', kernel_regularizer=l2(0.001)))\n\tmodel.add(MaxPooling2D((2, 2)))\n\tmodel.add(Conv2D(64, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same', kernel_regularizer=l2(0.001)))\n\tmodel.add(Conv2D(64, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same', kernel_regularizer=l2(0.001)))\n\tmodel.add(MaxPooling2D((2, 2)))\n\tmodel.add(Conv2D(128, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same', kernel_regularizer=l2(0.001)))\n\tmodel.add(Conv2D(128, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same', kernel_regularizer=l2(0.001)))\n\tmodel.add(MaxPooling2D((2, 2)))\n\tmodel.add(Flatten())\n\tmodel.add(Dense(128, activation='relu', kernel_initializer='he_uniform', kernel_regularizer=l2(0.001)))\n\tmodel.add(Dense(10, activation='softmax'))\n\t# compile model\n\topt = SGD(lr=0.001, momentum=0.9)\n\tmodel.compile(optimizer=opt, loss='categorical_crossentropy', metrics=['accuracy'])\n\treturn model\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n14\n15\n16\n17\n18\n19\n# define cnn model\ndef\ndefine_model\n(\n)\n:\nmodel\n=\nSequential\n(\n)\nmodel\n.\nadd\n(\nConv2D\n(\n32\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n,\nkernel_regularizer\n=\nl2\n(\n0.001\n)\n,\ninput_shape\n=\n(\n32\n,\n32\n,\n3\n)\n)\n)\nmodel\n.\nadd\n(\nConv2D\n(\n32\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n,\nkernel_regularizer\n=\nl2\n(\n0.001\n)\n)\n)\nmodel\n.\nadd\n(\nMaxPooling2D\n(\n(\n2\n,\n2\n)\n)\n)\nmodel\n.\nadd\n(\nConv2D\n(\n64\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n,\nkernel_regularizer\n=\nl2\n(\n0.001\n)\n)\n)\nmodel\n.\nadd\n(\nConv2D\n(\n64\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n,\nkernel_regularizer\n=\nl2\n(\n0.001\n)\n)\n)\nmodel\n.\nadd\n(\nMaxPooling2D\n(\n(\n2\n,\n2\n)\n)\n)\nmodel\n.\nadd\n(\nConv2D\n(\n128\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n,\nkernel_regularizer\n=\nl2\n(\n0.001\n)\n)\n)\nmodel\n.\nadd\n(\nConv2D\n(\n128\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n,\nkernel_regularizer\n=\nl2\n(\n0.001\n)\n)\n)\nmodel\n.\nadd\n(\nMaxPooling2D\n(\n(\n2\n,\n2\n)\n)\n)\nmodel\n.\nadd\n(\nFlatten\n(\n)\n)\nmodel\n.\nadd\n(\nDense\n(\n128\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\nkernel_regularizer\n=\nl2\n(\n0.001\n)\n)\n)\nmodel\n.\nadd\n(\nDense\n(\n10\n,\nactivation\n=\n'softmax'\n)\n)\n# compile model\nopt\n=\nSGD\n(\nlr\n=\n0.001\n,\nmomentum\n=\n0.9\n)\nmodel\n.\ncompile\n(\noptimizer\n=\nopt\n,\nloss\n=\n'categorical_crossentropy'\n,\nmetrics\n=\n[\n'accuracy'\n]\n)\nreturn\nmodel\nThe full code listing is provided below for completeness.\n# baseline model with weight decay on the cifar10 dataset\nimport sys\nfrom matplotlib import pyplot\nfrom keras.datasets import cifar10\nfrom keras.utils import to_categorical\nfrom keras.models import Sequential\nfrom keras.layers import Conv2D\nfrom keras.layers import MaxPooling2D\nfrom keras.layers import Dense\nfrom keras.layers import Flatten\nfrom keras.optimizers import SGD\nfrom keras.regularizers import l2\n\n# load train and test dataset\ndef load_dataset():\n\t# load dataset\n\t(trainX, trainY), (testX, testY) = cifar10.load_data()\n\t# one hot encode target values\n\ttrainY = to_categorical(trainY)\n\ttestY = to_categorical(testY)\n\treturn trainX, trainY, testX, testY\n\n# scale pixels\ndef prep_pixels(train, test):\n\t# convert from integers to floats\n\ttrain_norm = train.astype('float32')\n\ttest_norm = test.astype('float32')\n\t# normalize to range 0-1\n\ttrain_norm = train_norm / 255.0\n\ttest_norm = test_norm / 255.0\n\t# return normalized images\n\treturn train_norm, test_norm\n\n# define cnn model\ndef define_model():\n\tmodel = Sequential()\n\tmodel.add(Conv2D(32, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same', kernel_regularizer=l2(0.001), input_shape=(32, 32, 3)))\n\tmodel.add(Conv2D(32, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same', kernel_regularizer=l2(0.001)))\n\tmodel.add(MaxPooling2D((2, 2)))\n\tmodel.add(Conv2D(64, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same', kernel_regularizer=l2(0.001)))\n\tmodel.add(Conv2D(64, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same', kernel_regularizer=l2(0.001)))\n\tmodel.add(MaxPooling2D((2, 2)))\n\tmodel.add(Conv2D(128, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same', kernel_regularizer=l2(0.001)))\n\tmodel.add(Conv2D(128, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same', kernel_regularizer=l2(0.001)))\n\tmodel.add(MaxPooling2D((2, 2)))\n\tmodel.add(Flatten())\n\tmodel.add(Dense(128, activation='relu', kernel_initializer='he_uniform', kernel_regularizer=l2(0.001)))\n\tmodel.add(Dense(10, activation='softmax'))\n\t# compile model\n\topt = SGD(lr=0.001, momentum=0.9)\n\tmodel.compile(optimizer=opt, loss='categorical_crossentropy', metrics=['accuracy'])\n\treturn model\n\n# plot diagnostic learning curves\ndef summarize_diagnostics(history):\n\t# plot loss\n\tpyplot.subplot(211)\n\tpyplot.title('Cross Entropy Loss')\n\tpyplot.plot(history.history['loss'], color='blue', label='train')\n\tpyplot.plot(history.history['val_loss'], color='orange', label='test')\n\t# plot accuracy\n\tpyplot.subplot(212)\n\tpyplot.title('Classification Accuracy')\n\tpyplot.plot(history.history['accuracy'], color='blue', label='train')\n\tpyplot.plot(history.history['val_accuracy'], color='orange', label='test')\n\t# save plot to file\n\tfilename = sys.argv[0].split('/')[-1]\n\tpyplot.savefig(filename + '_plot.png')\n\tpyplot.close()\n\n# run the test harness for evaluating a model\ndef run_test_harness():\n\t# load dataset\n\ttrainX, trainY, testX, testY = load_dataset()\n\t# prepare pixel data\n\ttrainX, testX = prep_pixels(trainX, testX)\n\t# define model\n\tmodel = define_model()\n\t# fit model\n\thistory = model.fit(trainX, trainY, epochs=100, batch_size=64, validation_data=(testX, testY), verbose=0)\n\t# evaluate model\n\t_, acc = model.evaluate(testX, testY, verbose=0)\n\tprint('> %.3f' % (acc * 100.0))\n\t# learning curves\n\tsummarize_diagnostics(history)\n\n# entry point, run the test harness\nrun_test_harness()\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n14\n15\n16\n17\n18\n19\n20\n21\n22\n23\n24\n25\n26\n27\n28\n29\n30\n31\n32\n33\n34\n35\n36\n37\n38\n39\n40\n41\n42\n43\n44\n45\n46\n47\n48\n49\n50\n51\n52\n53\n54\n55\n56\n57\n58\n59\n60\n61\n62\n63\n64\n65\n66\n67\n68\n69\n70\n71\n72\n73\n74\n75\n76\n77\n78\n79\n80\n81\n82\n83\n84\n85\n86\n87\n88\n# baseline model with weight decay on the cifar10 dataset\nimport\nsys\nfrom\nmatplotlib\nimport\npyplot\nfrom\nkeras\n.\ndatasets\nimport\ncifar10\nfrom\nkeras\n.\nutils\nimport\nto_categorical\nfrom\nkeras\n.\nmodels\nimport\nSequential\nfrom\nkeras\n.\nlayers\nimport\nConv2D\nfrom\nkeras\n.\nlayers\nimport\nMaxPooling2D\nfrom\nkeras\n.\nlayers\nimport\nDense\nfrom\nkeras\n.\nlayers\nimport\nFlatten\nfrom\nkeras\n.\noptimizers\nimport\nSGD\nfrom\nkeras\n.\nregularizers\nimport\nl2\n# load train and test dataset\ndef\nload_dataset\n(\n)\n:\n# load dataset\n(\ntrainX\n,\ntrainY\n)\n,\n(\ntestX\n,\ntestY\n)\n=\ncifar10\n.\nload_data\n(\n)\n# one hot encode target values\ntrainY\n=\nto_categorical\n(\ntrainY\n)\ntestY\n=\nto_categorical\n(\ntestY\n)\nreturn\ntrainX\n,\ntrainY\n,\ntestX\n,\ntestY\n# scale pixels\ndef\nprep_pixels\n(\ntrain\n,\ntest\n)\n:\n# convert from integers to floats\ntrain_norm\n=\ntrain\n.\nastype\n(\n'float32'\n)\ntest_norm\n=\ntest\n.\nastype\n(\n'float32'\n)\n# normalize to range 0-1\ntrain_norm\n=\ntrain_norm\n/\n255.0\ntest_norm\n=\ntest_norm\n/\n255.0\n# return normalized images\nreturn\ntrain_norm\n,\ntest\n_\nnorm\n# define cnn model\ndef\ndefine_model\n(\n)\n:\nmodel\n=\nSequential\n(\n)\nmodel\n.\nadd\n(\nConv2D\n(\n32\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n,\nkernel_regularizer\n=\nl2\n(\n0.001\n)\n,\ninput_shape\n=\n(\n32\n,\n32\n,\n3\n)\n)\n)\nmodel\n.\nadd\n(\nConv2D\n(\n32\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n,\nkernel_regularizer\n=\nl2\n(\n0.001\n)\n)\n)\nmodel\n.\nadd\n(\nMaxPooling2D\n(\n(\n2\n,\n2\n)\n)\n)\nmodel\n.\nadd\n(\nConv2D\n(\n64\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n,\nkernel_regularizer\n=\nl2\n(\n0.001\n)\n)\n)\nmodel\n.\nadd\n(\nConv2D\n(\n64\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n,\nkernel_regularizer\n=\nl2\n(\n0.001\n)\n)\n)\nmodel\n.\nadd\n(\nMaxPooling2D\n(\n(\n2\n,\n2\n)\n)\n)\nmodel\n.\nadd\n(\nConv2D\n(\n128\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n,\nkernel_regularizer\n=\nl2\n(\n0.001\n)\n)\n)\nmodel\n.\nadd\n(\nConv2D\n(\n128\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n,\nkernel_regularizer\n=\nl2\n(\n0.001\n)\n)\n)\nmodel\n.\nadd\n(\nMaxPooling2D\n(\n(\n2\n,\n2\n)\n)\n)\nmodel\n.\nadd\n(\nFlatten\n(\n)\n)\nmodel\n.\nadd\n(\nDense\n(\n128\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\nkernel_regularizer\n=\nl2\n(\n0.001\n)\n)\n)\nmodel\n.\nadd\n(\nDense\n(\n10\n,\nactivation\n=\n'softmax'\n)\n)\n# compile model\nopt\n=\nSGD\n(\nlr\n=\n0.001\n,\nmomentum\n=\n0.9\n)\nmodel\n.\ncompile\n(\noptimizer\n=\nopt\n,\nloss\n=\n'categorical_crossentropy'\n,\nmetrics\n=\n[\n'accuracy'\n]\n)\nreturn\nmodel\n# plot diagnostic learning curves\ndef\nsummarize_diagnostics\n(\nhistory\n)\n:\n# plot loss\npyplot\n.\nsubplot\n(\n211\n)\npyplot\n.\ntitle\n(\n'Cross Entropy Loss'\n)\npyplot\n.\nplot\n(\nhistory\n.\nhistory\n[\n'loss'\n]\n,\ncolor\n=\n'blue'\n,\nlabel\n=\n'train'\n)\npyplot\n.\nplot\n(\nhistory\n.\nhistory\n[\n'val_loss'\n]\n,\ncolor\n=\n'orange'\n,\nlabel\n=\n'test'\n)\n# plot accuracy\npyplot\n.\nsubplot\n(\n212\n)\npyplot\n.\ntitle\n(\n'Classification Accuracy'\n)\npyplot\n.\nplot\n(\nhistory\n.\nhistory\n[\n'accuracy'\n]\n,\ncolor\n=\n'blue'\n,\nlabel\n=\n'train'\n)\npyplot\n.\nplot\n(\nhistory\n.\nhistory\n[\n'val_accuracy'\n]\n,\ncolor\n=\n'orange'\n,\nlabel\n=\n'test'\n)\n# save plot to file\nfilename\n=\nsys\n.\nargv\n[\n0\n]\n.\nsplit\n(\n'/'\n)\n[\n-\n1\n]\npyplot\n.\nsavefig\n(\nfilename\n+\n'_plot.png'\n)\npyplot\n.\nclose\n(\n)\n# run the test harness for evaluating a model\ndef\nrun_test_harness\n(\n)\n:\n# load dataset\ntrainX\n,\ntrainY\n,\ntestX\n,\ntestY\n=\nload_dataset\n(\n)\n# prepare pixel data\ntrainX\n,\ntestX\n=\nprep_pixels\n(\ntrainX\n,\ntestX\n)\n# define model\nmodel\n=\ndefine_model\n(\n)\n# fit model\nhistory\n=\nmodel\n.\nfit\n(\ntrainX\n,\ntrainY\n,\nepochs\n=\n100\n,\nbatch_size\n=\n64\n,\nvalidation_data\n=\n(\ntestX\n,\ntestY\n)\n,\nverbose\n=\n0\n)\n# evaluate model\n_\n,\nacc\n=\nmodel\n.\nevaluate\n(\ntestX\n,\ntestY\n,\nverbose\n=\n0\n)\nprint\n(\n'> %.3f'\n%\n(\nacc *\n100.0\n)\n)\n# learning curves\nsummarize_diagnostics\n(\nhistory\n)\n# entry point, run the test harness\nrun_test_harness\n(\n)\nRunning the model in the test harness prints the classification accuracy of the test dataset.\nNote\n: Your\nresults may vary\ngiven the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.\nIn this case, we see no improvement in the model performance on the test set; in fact, we see a small drop in performance from about 73% to about 72% classification accuracy.\n> 72.550\n1\n> 72.550\nReviewing the learning curves, we do see a small reduction in the overfitting, but the impact is not as effective as dropout.\nWe might be able to improve the effect of weight decay by perhaps using a larger weighting, such as 0.01 or even 0.1.\nLine Plots of Learning Curves for Baseline Model With Weight Decay on the CIFAR-10 Dataset\nData Augmentation\nData augmentation involves making copies of the examples in the training dataset with small random modifications.\nThis has a regularizing effect as it both expands the training dataset and allows the model to learn the same general features, although in a more generalized manner.\nThere are many types of data augmentation that could be applied. Given that the dataset is comprised of small photos of objects, we do not want to use augmentation that distorts the images too much, so that useful features in the images can be preserved and used.\nThe types of random augmentations that could be useful include a horizontal flip, minor shifts of the image, and perhaps small zooming or cropping of the image.\nWe will investigate the effect of simple augmentation on the baseline image, specifically horizontal flips and 10% shifts in the height and width of the image.\nThis can be implemented in Keras using the\nImageDataGenerator class\n; for example:\n# create data generator\ndatagen = ImageDataGenerator(width_shift_range=0.1, height_shift_range=0.1, horizontal_flip=True)\n# prepare iterator\nit_train = datagen.flow(trainX, trainY, batch_size=64)\n1\n2\n3\n4\n# create data generator\ndatagen\n=\nImageDataGenerator\n(\nwidth_shift_range\n=\n0.1\n,\nheight_shift_range\n=\n0.1\n,\nhorizontal_flip\n=\nTrue\n)\n# prepare iterator\nit_train\n=\ndatagen\n.\nflow\n(\ntrainX\n,\ntrainY\n,\nbatch_size\n=\n64\n)\nThis can be used during training by passing the iterator to the\nmodel.fit_generator()\nfunction and defining the number of batches in a single epoch.\n# fit model\nsteps = int(trainX.shape[0] / 64)\nhistory = model.fit_generator(it_train, steps_per_epoch=steps, epochs=100, validation_data=(testX, testY), verbose=0)\n1\n2\n3\n# fit model\nsteps\n=\nint\n(\ntrainX\n.\nshape\n[\n0\n]\n/\n64\n)\nhistory\n=\nmodel\n.\nfit_generator\n(\nit_train\n,\nsteps_per_epoch\n=\nsteps\n,\nepochs\n=\n100\n,\nvalidation_data\n=\n(\ntestX\n,\ntestY\n)\n,\nverbose\n=\n0\n)\nNo changes to the model are required.\nThe updated version of the\nrun_test_harness()\nfunction to support data augmentation is listed below.\n# run the test harness for evaluating a model\ndef run_test_harness():\n\t# load dataset\n\ttrainX, trainY, testX, testY = load_dataset()\n\t# prepare pixel data\n\ttrainX, testX = prep_pixels(trainX, testX)\n\t# define model\n\tmodel = define_model()\n\t# create data generator\n\tdatagen = ImageDataGenerator(width_shift_range=0.1, height_shift_range=0.1, horizontal_flip=True)\n\t# prepare iterator\n\tit_train = datagen.flow(trainX, trainY, batch_size=64)\n\t# fit model\n\tsteps = int(trainX.shape[0] / 64)\n\thistory = model.fit_generator(it_train, steps_per_epoch=steps, epochs=100, validation_data=(testX, testY), verbose=0)\n\t# evaluate model\n\t_, acc = model.evaluate(testX, testY, verbose=0)\n\tprint('> %.3f' % (acc * 100.0))\n\t# learning curves\n\tsummarize_diagnostics(history)\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n14\n15\n16\n17\n18\n19\n20\n# run the test harness for evaluating a model\ndef\nrun_test_harness\n(\n)\n:\n# load dataset\ntrainX\n,\ntrainY\n,\ntestX\n,\ntestY\n=\nload_dataset\n(\n)\n# prepare pixel data\ntrainX\n,\ntestX\n=\nprep_pixels\n(\ntrainX\n,\ntestX\n)\n# define model\nmodel\n=\ndefine_model\n(\n)\n# create data generator\ndatagen\n=\nImageDataGenerator\n(\nwidth_shift_range\n=\n0.1\n,\nheight_shift_range\n=\n0.1\n,\nhorizontal_flip\n=\nTrue\n)\n# prepare iterator\nit_train\n=\ndatagen\n.\nflow\n(\ntrainX\n,\ntrainY\n,\nbatch_size\n=\n64\n)\n# fit model\nsteps\n=\nint\n(\ntrainX\n.\nshape\n[\n0\n]\n/\n64\n)\nhistory\n=\nmodel\n.\nfit_generator\n(\nit_train\n,\nsteps_per_epoch\n=\nsteps\n,\nepochs\n=\n100\n,\nvalidation_data\n=\n(\ntestX\n,\ntestY\n)\n,\nverbose\n=\n0\n)\n# evaluate model\n_\n,\nacc\n=\nmodel\n.\nevaluate\n(\ntestX\n,\ntestY\n,\nverbose\n=\n0\n)\nprint\n(\n'> %.3f'\n%\n(\nacc *\n100.0\n)\n)\n# learning curves\nsummarize_diagnostics\n(\nhistory\n)\nThe full code listing is provided below for completeness.\n# baseline model with data augmentation on the cifar10 dataset\nimport sys\nfrom matplotlib import pyplot\nfrom keras.datasets import cifar10\nfrom keras.utils import to_categorical\nfrom keras.models import Sequential\nfrom keras.layers import Conv2D\nfrom keras.layers import MaxPooling2D\nfrom keras.layers import Dense\nfrom keras.layers import Flatten\nfrom keras.optimizers import SGD\nfrom keras.preprocessing.image import ImageDataGenerator\n\n# load train and test dataset\ndef load_dataset():\n\t# load dataset\n\t(trainX, trainY), (testX, testY) = cifar10.load_data()\n\t# one hot encode target values\n\ttrainY = to_categorical(trainY)\n\ttestY = to_categorical(testY)\n\treturn trainX, trainY, testX, testY\n\n# scale pixels\ndef prep_pixels(train, test):\n\t# convert from integers to floats\n\ttrain_norm = train.astype('float32')\n\ttest_norm = test.astype('float32')\n\t# normalize to range 0-1\n\ttrain_norm = train_norm / 255.0\n\ttest_norm = test_norm / 255.0\n\t# return normalized images\n\treturn train_norm, test_norm\n\n# define cnn model\ndef define_model():\n\tmodel = Sequential()\n\tmodel.add(Conv2D(32, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same', input_shape=(32, 32, 3)))\n\tmodel.add(Conv2D(32, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))\n\tmodel.add(MaxPooling2D((2, 2)))\n\tmodel.add(Conv2D(64, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))\n\tmodel.add(Conv2D(64, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))\n\tmodel.add(MaxPooling2D((2, 2)))\n\tmodel.add(Conv2D(128, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))\n\tmodel.add(Conv2D(128, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))\n\tmodel.add(MaxPooling2D((2, 2)))\n\tmodel.add(Flatten())\n\tmodel.add(Dense(128, activation='relu', kernel_initializer='he_uniform'))\n\tmodel.add(Dense(10, activation='softmax'))\n\t# compile model\n\topt = SGD(lr=0.001, momentum=0.9)\n\tmodel.compile(optimizer=opt, loss='categorical_crossentropy', metrics=['accuracy'])\n\treturn model\n\n# plot diagnostic learning curves\ndef summarize_diagnostics(history):\n\t# plot loss\n\tpyplot.subplot(211)\n\tpyplot.title('Cross Entropy Loss')\n\tpyplot.plot(history.history['loss'], color='blue', label='train')\n\tpyplot.plot(history.history['val_loss'], color='orange', label='test')\n\t# plot accuracy\n\tpyplot.subplot(212)\n\tpyplot.title('Classification Accuracy')\n\tpyplot.plot(history.history['accuracy'], color='blue', label='train')\n\tpyplot.plot(history.history['val_accuracy'], color='orange', label='test')\n\t# save plot to file\n\tfilename = sys.argv[0].split('/')[-1]\n\tpyplot.savefig(filename + '_plot.png')\n\tpyplot.close()\n\n# run the test harness for evaluating a model\ndef run_test_harness():\n\t# load dataset\n\ttrainX, trainY, testX, testY = load_dataset()\n\t# prepare pixel data\n\ttrainX, testX = prep_pixels(trainX, testX)\n\t# define model\n\tmodel = define_model()\n\t# create data generator\n\tdatagen = ImageDataGenerator(width_shift_range=0.1, height_shift_range=0.1, horizontal_flip=True)\n\t# prepare iterator\n\tit_train = datagen.flow(trainX, trainY, batch_size=64)\n\t# fit model\n\tsteps = int(trainX.shape[0] / 64)\n\thistory = model.fit_generator(it_train, steps_per_epoch=steps, epochs=100, validation_data=(testX, testY), verbose=0)\n\t# evaluate model\n\t_, acc = model.evaluate(testX, testY, verbose=0)\n\tprint('> %.3f' % (acc * 100.0))\n\t# learning curves\n\tsummarize_diagnostics(history)\n\n# entry point, run the test harness\nrun_test_harness()\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n14\n15\n16\n17\n18\n19\n20\n21\n22\n23\n24\n25\n26\n27\n28\n29\n30\n31\n32\n33\n34\n35\n36\n37\n38\n39\n40\n41\n42\n43\n44\n45\n46\n47\n48\n49\n50\n51\n52\n53\n54\n55\n56\n57\n58\n59\n60\n61\n62\n63\n64\n65\n66\n67\n68\n69\n70\n71\n72\n73\n74\n75\n76\n77\n78\n79\n80\n81\n82\n83\n84\n85\n86\n87\n88\n89\n90\n91\n92\n93\n# baseline model with data augmentation on the cifar10 dataset\nimport\nsys\nfrom\nmatplotlib\nimport\npyplot\nfrom\nkeras\n.\ndatasets\nimport\ncifar10\nfrom\nkeras\n.\nutils\nimport\nto_categorical\nfrom\nkeras\n.\nmodels\nimport\nSequential\nfrom\nkeras\n.\nlayers\nimport\nConv2D\nfrom\nkeras\n.\nlayers\nimport\nMaxPooling2D\nfrom\nkeras\n.\nlayers\nimport\nDense\nfrom\nkeras\n.\nlayers\nimport\nFlatten\nfrom\nkeras\n.\noptimizers\nimport\nSGD\nfrom\nkeras\n.\npreprocessing\n.\nimage\nimport\nImageDataGenerator\n# load train and test dataset\ndef\nload_dataset\n(\n)\n:\n# load dataset\n(\ntrainX\n,\ntrainY\n)\n,\n(\ntestX\n,\ntestY\n)\n=\ncifar10\n.\nload_data\n(\n)\n# one hot encode target values\ntrainY\n=\nto_categorical\n(\ntrainY\n)\ntestY\n=\nto_categorical\n(\ntestY\n)\nreturn\ntrainX\n,\ntrainY\n,\ntestX\n,\ntestY\n# scale pixels\ndef\nprep_pixels\n(\ntrain\n,\ntest\n)\n:\n# convert from integers to floats\ntrain_norm\n=\ntrain\n.\nastype\n(\n'float32'\n)\ntest_norm\n=\ntest\n.\nastype\n(\n'float32'\n)\n# normalize to range 0-1\ntrain_norm\n=\ntrain_norm\n/\n255.0\ntest_norm\n=\ntest_norm\n/\n255.0\n# return normalized images\nreturn\ntrain_norm\n,\ntest\n_\nnorm\n# define cnn model\ndef\ndefine_model\n(\n)\n:\nmodel\n=\nSequential\n(\n)\nmodel\n.\nadd\n(\nConv2D\n(\n32\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n,\ninput_shape\n=\n(\n32\n,\n32\n,\n3\n)\n)\n)\nmodel\n.\nadd\n(\nConv2D\n(\n32\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n)\n)\nmodel\n.\nadd\n(\nMaxPooling2D\n(\n(\n2\n,\n2\n)\n)\n)\nmodel\n.\nadd\n(\nConv2D\n(\n64\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n)\n)\nmodel\n.\nadd\n(\nConv2D\n(\n64\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n)\n)\nmodel\n.\nadd\n(\nMaxPooling2D\n(\n(\n2\n,\n2\n)\n)\n)\nmodel\n.\nadd\n(\nConv2D\n(\n128\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n)\n)\nmodel\n.\nadd\n(\nConv2D\n(\n128\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n)\n)\nmodel\n.\nadd\n(\nMaxPooling2D\n(\n(\n2\n,\n2\n)\n)\n)\nmodel\n.\nadd\n(\nFlatten\n(\n)\n)\nmodel\n.\nadd\n(\nDense\n(\n128\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n)\n)\nmodel\n.\nadd\n(\nDense\n(\n10\n,\nactivation\n=\n'softmax'\n)\n)\n# compile model\nopt\n=\nSGD\n(\nlr\n=\n0.001\n,\nmomentum\n=\n0.9\n)\nmodel\n.\ncompile\n(\noptimizer\n=\nopt\n,\nloss\n=\n'categorical_crossentropy'\n,\nmetrics\n=\n[\n'accuracy'\n]\n)\nreturn\nmodel\n# plot diagnostic learning curves\ndef\nsummarize_diagnostics\n(\nhistory\n)\n:\n# plot loss\npyplot\n.\nsubplot\n(\n211\n)\npyplot\n.\ntitle\n(\n'Cross Entropy Loss'\n)\npyplot\n.\nplot\n(\nhistory\n.\nhistory\n[\n'loss'\n]\n,\ncolor\n=\n'blue'\n,\nlabel\n=\n'train'\n)\npyplot\n.\nplot\n(\nhistory\n.\nhistory\n[\n'val_loss'\n]\n,\ncolor\n=\n'orange'\n,\nlabel\n=\n'test'\n)\n# plot accuracy\npyplot\n.\nsubplot\n(\n212\n)\npyplot\n.\ntitle\n(\n'Classification Accuracy'\n)\npyplot\n.\nplot\n(\nhistory\n.\nhistory\n[\n'accuracy'\n]\n,\ncolor\n=\n'blue'\n,\nlabel\n=\n'train'\n)\npyplot\n.\nplot\n(\nhistory\n.\nhistory\n[\n'val_accuracy'\n]\n,\ncolor\n=\n'orange'\n,\nlabel\n=\n'test'\n)\n# save plot to file\nfilename\n=\nsys\n.\nargv\n[\n0\n]\n.\nsplit\n(\n'/'\n)\n[\n-\n1\n]\npyplot\n.\nsavefig\n(\nfilename\n+\n'_plot.png'\n)\npyplot\n.\nclose\n(\n)\n# run the test harness for evaluating a model\ndef\nrun_test_harness\n(\n)\n:\n# load dataset\ntrainX\n,\ntrainY\n,\ntestX\n,\ntestY\n=\nload_dataset\n(\n)\n# prepare pixel data\ntrainX\n,\ntestX\n=\nprep_pixels\n(\ntrainX\n,\ntestX\n)\n# define model\nmodel\n=\ndefine_model\n(\n)\n# create data generator\ndatagen\n=\nImageDataGenerator\n(\nwidth_shift_range\n=\n0.1\n,\nheight_shift_range\n=\n0.1\n,\nhorizontal_flip\n=\nTrue\n)\n# prepare iterator\nit_train\n=\ndatagen\n.\nflow\n(\ntrainX\n,\ntrainY\n,\nbatch_size\n=\n64\n)\n# fit model\nsteps\n=\nint\n(\ntrainX\n.\nshape\n[\n0\n]\n/\n64\n)\nhistory\n=\nmodel\n.\nfit_generator\n(\nit_train\n,\nsteps_per_epoch\n=\nsteps\n,\nepochs\n=\n100\n,\nvalidation_data\n=\n(\ntestX\n,\ntestY\n)\n,\nverbose\n=\n0\n)\n# evaluate model\n_\n,\nacc\n=\nmodel\n.\nevaluate\n(\ntestX\n,\ntestY\n,\nverbose\n=\n0\n)\nprint\n(\n'> %.3f'\n%\n(\nacc *\n100.0\n)\n)\n# learning curves\nsummarize_diagnostics\n(\nhistory\n)\n# entry point, run the test harness\nrun_test_harness\n(\n)\nRunning the model in the test harness prints the classification accuracy on the test dataset.\nNote\n: Your\nresults may vary\ngiven the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.\nIn this case, we see another large improvement in model performance, much like we saw with dropout. In this case, an improvement of about 11% from about 73% for the baseline model to about 84%.\n> 84.470\n1\n> 84.470\nReviewing the learning curves, we see a similar improvement in model performances as we do with dropout, although the plot of loss suggests that model performance on the test set may have stalled slightly sooner than it did with dropout.\nThe results suggest that perhaps a configuration that used both dropout and data augmentation might be effective.\nLine Plots of Learning Curves for Baseline Model With Data Augmentation on the CIFAR-10 Dataset\nDiscussion\nIn this section, we explored three approaches designed to slow down the convergence of the model.\nA summary of the results is provided below:\nBaseline + Dropout\n: 83.450%\nBaseline + Weight Decay\n: 72.550%\nBaseline + Data Augmentation\n: 84.470%\nThe results suggest that both dropout and data augmentation are having the desired effect, and weight decay, at least for the chosen configuration, did not.\nNow that the model is learning well, we can look for both improvements on what is working, as well as combinations on what is working.\nHow to Develop Further Improvements\nIn the previous section, we discovered that dropout and data augmentation, when added to the baseline model, result in a model that learns the problem well.\nWe will now investigate refinements of these techniques to see if we can further improve the model’s performance. Specifically, we will look at a variation of dropout regularization and combining dropout with data augmentation.\nLearning has slowed down, so we will investigate increasing the number of training epochs to give the model enough space, if needed, to expose the learning dynamics in the learning curves.\nVariation of Dropout Regularization\nDropout is working very well, so it may be worth investigating variations of how dropout is applied to the model.\nOne variation that might be interesting is to increase the amount of dropout from 20% to 25% or 30%. Another variation that might be interesting is using a pattern of increasing dropout from 20% for the first block, 30% for the second block, and so on to 50% at the fully connected layer in the classifier part of the model.\nThis type of increasing dropout with the depth of the model is a common pattern. It is effective as it forces layers deep in the model to regularize more than layers closer to the input.\nThe baseline model with dropout updated to use a pattern of increasing dropout percentage with model depth is defined below.\n# define cnn model\ndef define_model():\n\tmodel = Sequential()\n\tmodel.add(Conv2D(32, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same', input_shape=(32, 32, 3)))\n\tmodel.add(Conv2D(32, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))\n\tmodel.add(MaxPooling2D((2, 2)))\n\tmodel.add(Dropout(0.2))\n\tmodel.add(Conv2D(64, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))\n\tmodel.add(Conv2D(64, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))\n\tmodel.add(MaxPooling2D((2, 2)))\n\tmodel.add(Dropout(0.3))\n\tmodel.add(Conv2D(128, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))\n\tmodel.add(Conv2D(128, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))\n\tmodel.add(MaxPooling2D((2, 2)))\n\tmodel.add(Dropout(0.4))\n\tmodel.add(Flatten())\n\tmodel.add(Dense(128, activation='relu', kernel_initializer='he_uniform'))\n\tmodel.add(Dropout(0.5))\n\tmodel.add(Dense(10, activation='softmax'))\n\t# compile model\n\topt = SGD(lr=0.001, momentum=0.9)\n\tmodel.compile(optimizer=opt, loss='categorical_crossentropy', metrics=['accuracy'])\n\treturn model\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n14\n15\n16\n17\n18\n19\n20\n21\n22\n23\n# define cnn model\ndef\ndefine_model\n(\n)\n:\nmodel\n=\nSequential\n(\n)\nmodel\n.\nadd\n(\nConv2D\n(\n32\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n,\ninput_shape\n=\n(\n32\n,\n32\n,\n3\n)\n)\n)\nmodel\n.\nadd\n(\nConv2D\n(\n32\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n)\n)\nmodel\n.\nadd\n(\nMaxPooling2D\n(\n(\n2\n,\n2\n)\n)\n)\nmodel\n.\nadd\n(\nDropout\n(\n0.2\n)\n)\nmodel\n.\nadd\n(\nConv2D\n(\n64\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n)\n)\nmodel\n.\nadd\n(\nConv2D\n(\n64\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n)\n)\nmodel\n.\nadd\n(\nMaxPooling2D\n(\n(\n2\n,\n2\n)\n)\n)\nmodel\n.\nadd\n(\nDropout\n(\n0.3\n)\n)\nmodel\n.\nadd\n(\nConv2D\n(\n128\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n)\n)\nmodel\n.\nadd\n(\nConv2D\n(\n128\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n)\n)\nmodel\n.\nadd\n(\nMaxPooling2D\n(\n(\n2\n,\n2\n)\n)\n)\nmodel\n.\nadd\n(\nDropout\n(\n0.4\n)\n)\nmodel\n.\nadd\n(\nFlatten\n(\n)\n)\nmodel\n.\nadd\n(\nDense\n(\n128\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n)\n)\nmodel\n.\nadd\n(\nDropout\n(\n0.5\n)\n)\nmodel\n.\nadd\n(\nDense\n(\n10\n,\nactivation\n=\n'softmax'\n)\n)\n# compile model\nopt\n=\nSGD\n(\nlr\n=\n0.001\n,\nmomentum\n=\n0.9\n)\nmodel\n.\ncompile\n(\noptimizer\n=\nopt\n,\nloss\n=\n'categorical_crossentropy'\n,\nmetrics\n=\n[\n'accuracy'\n]\n)\nreturn\nmodel\nThe full code listing with this change is provided below for completeness.\n# baseline model with increasing dropout on the cifar10 dataset\nimport sys\nfrom matplotlib import pyplot\nfrom keras.datasets import cifar10\nfrom keras.utils import to_categorical\nfrom keras.models import Sequential\nfrom keras.layers import Conv2D\nfrom keras.layers import MaxPooling2D\nfrom keras.layers import Dense\nfrom keras.layers import Flatten\nfrom keras.layers import Dropout\nfrom keras.optimizers import SGD\n\n# load train and test dataset\ndef load_dataset():\n\t# load dataset\n\t(trainX, trainY), (testX, testY) = cifar10.load_data()\n\t# one hot encode target values\n\ttrainY = to_categorical(trainY)\n\ttestY = to_categorical(testY)\n\treturn trainX, trainY, testX, testY\n\n# scale pixels\ndef prep_pixels(train, test):\n\t# convert from integers to floats\n\ttrain_norm = train.astype('float32')\n\ttest_norm = test.astype('float32')\n\t# normalize to range 0-1\n\ttrain_norm = train_norm / 255.0\n\ttest_norm = test_norm / 255.0\n\t# return normalized images\n\treturn train_norm, test_norm\n\n# define cnn model\ndef define_model():\n\tmodel = Sequential()\n\tmodel.add(Conv2D(32, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same', input_shape=(32, 32, 3)))\n\tmodel.add(Conv2D(32, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))\n\tmodel.add(MaxPooling2D((2, 2)))\n\tmodel.add(Dropout(0.2))\n\tmodel.add(Conv2D(64, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))\n\tmodel.add(Conv2D(64, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))\n\tmodel.add(MaxPooling2D((2, 2)))\n\tmodel.add(Dropout(0.3))\n\tmodel.add(Conv2D(128, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))\n\tmodel.add(Conv2D(128, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))\n\tmodel.add(MaxPooling2D((2, 2)))\n\tmodel.add(Dropout(0.4))\n\tmodel.add(Flatten())\n\tmodel.add(Dense(128, activation='relu', kernel_initializer='he_uniform'))\n\tmodel.add(Dropout(0.5))\n\tmodel.add(Dense(10, activation='softmax'))\n\t# compile model\n\topt = SGD(lr=0.001, momentum=0.9)\n\tmodel.compile(optimizer=opt, loss='categorical_crossentropy', metrics=['accuracy'])\n\treturn model\n\n# plot diagnostic learning curves\ndef summarize_diagnostics(history):\n\t# plot loss\n\tpyplot.subplot(211)\n\tpyplot.title('Cross Entropy Loss')\n\tpyplot.plot(history.history['loss'], color='blue', label='train')\n\tpyplot.plot(history.history['val_loss'], color='orange', label='test')\n\t# plot accuracy\n\tpyplot.subplot(212)\n\tpyplot.title('Classification Accuracy')\n\tpyplot.plot(history.history['accuracy'], color='blue', label='train')\n\tpyplot.plot(history.history['val_accuracy'], color='orange', label='test')\n\t# save plot to file\n\tfilename = sys.argv[0].split('/')[-1]\n\tpyplot.savefig(filename + '_plot.png')\n\tpyplot.close()\n\n# run the test harness for evaluating a model\ndef run_test_harness():\n\t# load dataset\n\ttrainX, trainY, testX, testY = load_dataset()\n\t# prepare pixel data\n\ttrainX, testX = prep_pixels(trainX, testX)\n\t# define model\n\tmodel = define_model()\n\t# fit model\n\thistory = model.fit(trainX, trainY, epochs=200, batch_size=64, validation_data=(testX, testY), verbose=0)\n\t# evaluate model\n\t_, acc = model.evaluate(testX, testY, verbose=0)\n\tprint('> %.3f' % (acc * 100.0))\n\t# learning curves\n\tsummarize_diagnostics(history)\n\n# entry point, run the test harness\nrun_test_harness()\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n14\n15\n16\n17\n18\n19\n20\n21\n22\n23\n24\n25\n26\n27\n28\n29\n30\n31\n32\n33\n34\n35\n36\n37\n38\n39\n40\n41\n42\n43\n44\n45\n46\n47\n48\n49\n50\n51\n52\n53\n54\n55\n56\n57\n58\n59\n60\n61\n62\n63\n64\n65\n66\n67\n68\n69\n70\n71\n72\n73\n74\n75\n76\n77\n78\n79\n80\n81\n82\n83\n84\n85\n86\n87\n88\n89\n90\n91\n92\n# baseline model with increasing dropout on the cifar10 dataset\nimport\nsys\nfrom\nmatplotlib\nimport\npyplot\nfrom\nkeras\n.\ndatasets\nimport\ncifar10\nfrom\nkeras\n.\nutils\nimport\nto_categorical\nfrom\nkeras\n.\nmodels\nimport\nSequential\nfrom\nkeras\n.\nlayers\nimport\nConv2D\nfrom\nkeras\n.\nlayers\nimport\nMaxPooling2D\nfrom\nkeras\n.\nlayers\nimport\nDense\nfrom\nkeras\n.\nlayers\nimport\nFlatten\nfrom\nkeras\n.\nlayers\nimport\nDropout\nfrom\nkeras\n.\noptimizers\nimport\nSGD\n# load train and test dataset\ndef\nload_dataset\n(\n)\n:\n# load dataset\n(\ntrainX\n,\ntrainY\n)\n,\n(\ntestX\n,\ntestY\n)\n=\ncifar10\n.\nload_data\n(\n)\n# one hot encode target values\ntrainY\n=\nto_categorical\n(\ntrainY\n)\ntestY\n=\nto_categorical\n(\ntestY\n)\nreturn\ntrainX\n,\ntrainY\n,\ntestX\n,\ntestY\n# scale pixels\ndef\nprep_pixels\n(\ntrain\n,\ntest\n)\n:\n# convert from integers to floats\ntrain_norm\n=\ntrain\n.\nastype\n(\n'float32'\n)\ntest_norm\n=\ntest\n.\nastype\n(\n'float32'\n)\n# normalize to range 0-1\ntrain_norm\n=\ntrain_norm\n/\n255.0\ntest_norm\n=\ntest_norm\n/\n255.0\n# return normalized images\nreturn\ntrain_norm\n,\ntest\n_\nnorm\n# define cnn model\ndef\ndefine_model\n(\n)\n:\nmodel\n=\nSequential\n(\n)\nmodel\n.\nadd\n(\nConv2D\n(\n32\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n,\ninput_shape\n=\n(\n32\n,\n32\n,\n3\n)\n)\n)\nmodel\n.\nadd\n(\nConv2D\n(\n32\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n)\n)\nmodel\n.\nadd\n(\nMaxPooling2D\n(\n(\n2\n,\n2\n)\n)\n)\nmodel\n.\nadd\n(\nDropout\n(\n0.2\n)\n)\nmodel\n.\nadd\n(\nConv2D\n(\n64\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n)\n)\nmodel\n.\nadd\n(\nConv2D\n(\n64\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n)\n)\nmodel\n.\nadd\n(\nMaxPooling2D\n(\n(\n2\n,\n2\n)\n)\n)\nmodel\n.\nadd\n(\nDropout\n(\n0.3\n)\n)\nmodel\n.\nadd\n(\nConv2D\n(\n128\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n)\n)\nmodel\n.\nadd\n(\nConv2D\n(\n128\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n)\n)\nmodel\n.\nadd\n(\nMaxPooling2D\n(\n(\n2\n,\n2\n)\n)\n)\nmodel\n.\nadd\n(\nDropout\n(\n0.4\n)\n)\nmodel\n.\nadd\n(\nFlatten\n(\n)\n)\nmodel\n.\nadd\n(\nDense\n(\n128\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n)\n)\nmodel\n.\nadd\n(\nDropout\n(\n0.5\n)\n)\nmodel\n.\nadd\n(\nDense\n(\n10\n,\nactivation\n=\n'softmax'\n)\n)\n# compile model\nopt\n=\nSGD\n(\nlr\n=\n0.001\n,\nmomentum\n=\n0.9\n)\nmodel\n.\ncompile\n(\noptimizer\n=\nopt\n,\nloss\n=\n'categorical_crossentropy'\n,\nmetrics\n=\n[\n'accuracy'\n]\n)\nreturn\nmodel\n# plot diagnostic learning curves\ndef\nsummarize_diagnostics\n(\nhistory\n)\n:\n# plot loss\npyplot\n.\nsubplot\n(\n211\n)\npyplot\n.\ntitle\n(\n'Cross Entropy Loss'\n)\npyplot\n.\nplot\n(\nhistory\n.\nhistory\n[\n'loss'\n]\n,\ncolor\n=\n'blue'\n,\nlabel\n=\n'train'\n)\npyplot\n.\nplot\n(\nhistory\n.\nhistory\n[\n'val_loss'\n]\n,\ncolor\n=\n'orange'\n,\nlabel\n=\n'test'\n)\n# plot accuracy\npyplot\n.\nsubplot\n(\n212\n)\npyplot\n.\ntitle\n(\n'Classification Accuracy'\n)\npyplot\n.\nplot\n(\nhistory\n.\nhistory\n[\n'accuracy'\n]\n,\ncolor\n=\n'blue'\n,\nlabel\n=\n'train'\n)\npyplot\n.\nplot\n(\nhistory\n.\nhistory\n[\n'val_accuracy'\n]\n,\ncolor\n=\n'orange'\n,\nlabel\n=\n'test'\n)\n# save plot to file\nfilename\n=\nsys\n.\nargv\n[\n0\n]\n.\nsplit\n(\n'/'\n)\n[\n-\n1\n]\npyplot\n.\nsavefig\n(\nfilename\n+\n'_plot.png'\n)\npyplot\n.\nclose\n(\n)\n# run the test harness for evaluating a model\ndef\nrun_test_harness\n(\n)\n:\n# load dataset\ntrainX\n,\ntrainY\n,\ntestX\n,\ntestY\n=\nload_dataset\n(\n)\n# prepare pixel data\ntrainX\n,\ntestX\n=\nprep_pixels\n(\ntrainX\n,\ntestX\n)\n# define model\nmodel\n=\ndefine_model\n(\n)\n# fit model\nhistory\n=\nmodel\n.\nfit\n(\ntrainX\n,\ntrainY\n,\nepochs\n=\n200\n,\nbatch_size\n=\n64\n,\nvalidation_data\n=\n(\ntestX\n,\ntestY\n)\n,\nverbose\n=\n0\n)\n# evaluate model\n_\n,\nacc\n=\nmodel\n.\nevaluate\n(\ntestX\n,\ntestY\n,\nverbose\n=\n0\n)\nprint\n(\n'> %.3f'\n%\n(\nacc *\n100.0\n)\n)\n# learning curves\nsummarize_diagnostics\n(\nhistory\n)\n# entry point, run the test harness\nrun_test_harness\n(\n)\nRunning the model in the test harness prints the classification accuracy on the test dataset.\nNote\n: Your\nresults may vary\ngiven the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.\nIn this case, we can see a modest lift in performance from fixed dropout at about 83% to increasing dropout at about 84%.\n> 84.690\n1\n> 84.690\nReviewing the learning curves, we can see that the model converges well, with performance on the test dataset perhaps stalling at around 110 to 125 epochs. Compared to the learning curves for fixed dropout, we can see that again the rate of learning has been further slowed, allowing further refinement of the model without overfitting.\nThis is a fruitful area for investigation on this model, and perhaps more dropout layers and/or more aggressive dropout may result in further improvements.\nLine Plots of Learning Curves for Baseline Model With Increasing Dropout on the CIFAR-10 Dataset\nDropout and Data Augmentation\nIn the previous section, we discovered that both dropout and data augmentation resulted in a significant improvement in model performance.\nIn this section, we can experiment with combining both of these changes to the model to see if a further improvement can be achieved. Specifically, whether using both regularization techniques together results in better performance than either technique used alone.\nThe full code listing of a model with fixed dropout and data augmentation is provided below for completeness.\n# baseline model with dropout and data augmentation on the cifar10 dataset\nimport sys\nfrom matplotlib import pyplot\nfrom keras.datasets import cifar10\nfrom keras.utils import to_categorical\nfrom keras.models import Sequential\nfrom keras.layers import Conv2D\nfrom keras.layers import MaxPooling2D\nfrom keras.layers import Dense\nfrom keras.layers import Flatten\nfrom keras.optimizers import SGD\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom keras.layers import Dropout\n\n# load train and test dataset\ndef load_dataset():\n\t# load dataset\n\t(trainX, trainY), (testX, testY) = cifar10.load_data()\n\t# one hot encode target values\n\ttrainY = to_categorical(trainY)\n\ttestY = to_categorical(testY)\n\treturn trainX, trainY, testX, testY\n\n# scale pixels\ndef prep_pixels(train, test):\n\t# convert from integers to floats\n\ttrain_norm = train.astype('float32')\n\ttest_norm = test.astype('float32')\n\t# normalize to range 0-1\n\ttrain_norm = train_norm / 255.0\n\ttest_norm = test_norm / 255.0\n\t# return normalized images\n\treturn train_norm, test_norm\n\n# define cnn model\ndef define_model():\n\tmodel = Sequential()\n\tmodel.add(Conv2D(32, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same', input_shape=(32, 32, 3)))\n\tmodel.add(Conv2D(32, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))\n\tmodel.add(MaxPooling2D((2, 2)))\n\tmodel.add(Dropout(0.2))\n\tmodel.add(Conv2D(64, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))\n\tmodel.add(Conv2D(64, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))\n\tmodel.add(MaxPooling2D((2, 2)))\n\tmodel.add(Dropout(0.2))\n\tmodel.add(Conv2D(128, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))\n\tmodel.add(Conv2D(128, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))\n\tmodel.add(MaxPooling2D((2, 2)))\n\tmodel.add(Dropout(0.2))\n\tmodel.add(Flatten())\n\tmodel.add(Dense(128, activation='relu', kernel_initializer='he_uniform'))\n\tmodel.add(Dropout(0.2))\n\tmodel.add(Dense(10, activation='softmax'))\n\t# compile model\n\topt = SGD(lr=0.001, momentum=0.9)\n\tmodel.compile(optimizer=opt, loss='categorical_crossentropy', metrics=['accuracy'])\n\treturn model\n\n# plot diagnostic learning curves\ndef summarize_diagnostics(history):\n\t# plot loss\n\tpyplot.subplot(211)\n\tpyplot.title('Cross Entropy Loss')\n\tpyplot.plot(history.history['loss'], color='blue', label='train')\n\tpyplot.plot(history.history['val_loss'], color='orange', label='test')\n\t# plot accuracy\n\tpyplot.subplot(212)\n\tpyplot.title('Classification Accuracy')\n\tpyplot.plot(history.history['accuracy'], color='blue', label='train')\n\tpyplot.plot(history.history['val_accuracy'], color='orange', label='test')\n\t# save plot to file\n\tfilename = sys.argv[0].split('/')[-1]\n\tpyplot.savefig(filename + '_plot.png')\n\tpyplot.close()\n\n# run the test harness for evaluating a model\ndef run_test_harness():\n\t# load dataset\n\ttrainX, trainY, testX, testY = load_dataset()\n\t# prepare pixel data\n\ttrainX, testX = prep_pixels(trainX, testX)\n\t# define model\n\tmodel = define_model()\n\t# create data generator\n\tdatagen = ImageDataGenerator(width_shift_range=0.1, height_shift_range=0.1, horizontal_flip=True)\n\t# prepare iterator\n\tit_train = datagen.flow(trainX, trainY, batch_size=64)\n\t# fit model\n\tsteps = int(trainX.shape[0] / 64)\n\thistory = model.fit_generator(it_train, steps_per_epoch=steps, epochs=200, validation_data=(testX, testY), verbose=0)\n\t# evaluate model\n\t_, acc = model.evaluate(testX, testY, verbose=0)\n\tprint('> %.3f' % (acc * 100.0))\n\t# learning curves\n\tsummarize_diagnostics(history)\n\n# entry point, run the test harness\nrun_test_harness()\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n14\n15\n16\n17\n18\n19\n20\n21\n22\n23\n24\n25\n26\n27\n28\n29\n30\n31\n32\n33\n34\n35\n36\n37\n38\n39\n40\n41\n42\n43\n44\n45\n46\n47\n48\n49\n50\n51\n52\n53\n54\n55\n56\n57\n58\n59\n60\n61\n62\n63\n64\n65\n66\n67\n68\n69\n70\n71\n72\n73\n74\n75\n76\n77\n78\n79\n80\n81\n82\n83\n84\n85\n86\n87\n88\n89\n90\n91\n92\n93\n94\n95\n96\n97\n98\n# baseline model with dropout and data augmentation on the cifar10 dataset\nimport\nsys\nfrom\nmatplotlib\nimport\npyplot\nfrom\nkeras\n.\ndatasets\nimport\ncifar10\nfrom\nkeras\n.\nutils\nimport\nto_categorical\nfrom\nkeras\n.\nmodels\nimport\nSequential\nfrom\nkeras\n.\nlayers\nimport\nConv2D\nfrom\nkeras\n.\nlayers\nimport\nMaxPooling2D\nfrom\nkeras\n.\nlayers\nimport\nDense\nfrom\nkeras\n.\nlayers\nimport\nFlatten\nfrom\nkeras\n.\noptimizers\nimport\nSGD\nfrom\nkeras\n.\npreprocessing\n.\nimage\nimport\nImageDataGenerator\nfrom\nkeras\n.\nlayers\nimport\nDropout\n# load train and test dataset\ndef\nload_dataset\n(\n)\n:\n# load dataset\n(\ntrainX\n,\ntrainY\n)\n,\n(\ntestX\n,\ntestY\n)\n=\ncifar10\n.\nload_data\n(\n)\n# one hot encode target values\ntrainY\n=\nto_categorical\n(\ntrainY\n)\ntestY\n=\nto_categorical\n(\ntestY\n)\nreturn\ntrainX\n,\ntrainY\n,\ntestX\n,\ntestY\n# scale pixels\ndef\nprep_pixels\n(\ntrain\n,\ntest\n)\n:\n# convert from integers to floats\ntrain_norm\n=\ntrain\n.\nastype\n(\n'float32'\n)\ntest_norm\n=\ntest\n.\nastype\n(\n'float32'\n)\n# normalize to range 0-1\ntrain_norm\n=\ntrain_norm\n/\n255.0\ntest_norm\n=\ntest_norm\n/\n255.0\n# return normalized images\nreturn\ntrain_norm\n,\ntest\n_\nnorm\n# define cnn model\ndef\ndefine_model\n(\n)\n:\nmodel\n=\nSequential\n(\n)\nmodel\n.\nadd\n(\nConv2D\n(\n32\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n,\ninput_shape\n=\n(\n32\n,\n32\n,\n3\n)\n)\n)\nmodel\n.\nadd\n(\nConv2D\n(\n32\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n)\n)\nmodel\n.\nadd\n(\nMaxPooling2D\n(\n(\n2\n,\n2\n)\n)\n)\nmodel\n.\nadd\n(\nDropout\n(\n0.2\n)\n)\nmodel\n.\nadd\n(\nConv2D\n(\n64\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n)\n)\nmodel\n.\nadd\n(\nConv2D\n(\n64\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n)\n)\nmodel\n.\nadd\n(\nMaxPooling2D\n(\n(\n2\n,\n2\n)\n)\n)\nmodel\n.\nadd\n(\nDropout\n(\n0.2\n)\n)\nmodel\n.\nadd\n(\nConv2D\n(\n128\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n)\n)\nmodel\n.\nadd\n(\nConv2D\n(\n128\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n)\n)\nmodel\n.\nadd\n(\nMaxPooling2D\n(\n(\n2\n,\n2\n)\n)\n)\nmodel\n.\nadd\n(\nDropout\n(\n0.2\n)\n)\nmodel\n.\nadd\n(\nFlatten\n(\n)\n)\nmodel\n.\nadd\n(\nDense\n(\n128\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n)\n)\nmodel\n.\nadd\n(\nDropout\n(\n0.2\n)\n)\nmodel\n.\nadd\n(\nDense\n(\n10\n,\nactivation\n=\n'softmax'\n)\n)\n# compile model\nopt\n=\nSGD\n(\nlr\n=\n0.001\n,\nmomentum\n=\n0.9\n)\nmodel\n.\ncompile\n(\noptimizer\n=\nopt\n,\nloss\n=\n'categorical_crossentropy'\n,\nmetrics\n=\n[\n'accuracy'\n]\n)\nreturn\nmodel\n# plot diagnostic learning curves\ndef\nsummarize_diagnostics\n(\nhistory\n)\n:\n# plot loss\npyplot\n.\nsubplot\n(\n211\n)\npyplot\n.\ntitle\n(\n'Cross Entropy Loss'\n)\npyplot\n.\nplot\n(\nhistory\n.\nhistory\n[\n'loss'\n]\n,\ncolor\n=\n'blue'\n,\nlabel\n=\n'train'\n)\npyplot\n.\nplot\n(\nhistory\n.\nhistory\n[\n'val_loss'\n]\n,\ncolor\n=\n'orange'\n,\nlabel\n=\n'test'\n)\n# plot accuracy\npyplot\n.\nsubplot\n(\n212\n)\npyplot\n.\ntitle\n(\n'Classification Accuracy'\n)\npyplot\n.\nplot\n(\nhistory\n.\nhistory\n[\n'accuracy'\n]\n,\ncolor\n=\n'blue'\n,\nlabel\n=\n'train'\n)\npyplot\n.\nplot\n(\nhistory\n.\nhistory\n[\n'val_accuracy'\n]\n,\ncolor\n=\n'orange'\n,\nlabel\n=\n'test'\n)\n# save plot to file\nfilename\n=\nsys\n.\nargv\n[\n0\n]\n.\nsplit\n(\n'/'\n)\n[\n-\n1\n]\npyplot\n.\nsavefig\n(\nfilename\n+\n'_plot.png'\n)\npyplot\n.\nclose\n(\n)\n# run the test harness for evaluating a model\ndef\nrun_test_harness\n(\n)\n:\n# load dataset\ntrainX\n,\ntrainY\n,\ntestX\n,\ntestY\n=\nload_dataset\n(\n)\n# prepare pixel data\ntrainX\n,\ntestX\n=\nprep_pixels\n(\ntrainX\n,\ntestX\n)\n# define model\nmodel\n=\ndefine_model\n(\n)\n# create data generator\ndatagen\n=\nImageDataGenerator\n(\nwidth_shift_range\n=\n0.1\n,\nheight_shift_range\n=\n0.1\n,\nhorizontal_flip\n=\nTrue\n)\n# prepare iterator\nit_train\n=\ndatagen\n.\nflow\n(\ntrainX\n,\ntrainY\n,\nbatch_size\n=\n64\n)\n# fit model\nsteps\n=\nint\n(\ntrainX\n.\nshape\n[\n0\n]\n/\n64\n)\nhistory\n=\nmodel\n.\nfit_generator\n(\nit_train\n,\nsteps_per_epoch\n=\nsteps\n,\nepochs\n=\n200\n,\nvalidation_data\n=\n(\ntestX\n,\ntestY\n)\n,\nverbose\n=\n0\n)\n# evaluate model\n_\n,\nacc\n=\nmodel\n.\nevaluate\n(\ntestX\n,\ntestY\n,\nverbose\n=\n0\n)\nprint\n(\n'> %.3f'\n%\n(\nacc *\n100.0\n)\n)\n# learning curves\nsummarize_diagnostics\n(\nhistory\n)\n# entry point, run the test harness\nrun_test_harness\n(\n)\nRunning the model in the test harness prints the classification accuracy on the test dataset.\nNote\n: Your\nresults may vary\ngiven the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.\nIn this case, we can see that as we would have hoped, using both regularization techniques together has resulted in a further lift in model performance on the test set. In this case, combining fixed dropout with about 83% and data augmentation with about 84% has resulted in an improvement to about 85% classification accuracy.\n> 85.880\n1\n> 85.880\nReviewing the learning curves, we can see that the convergence behavior of the model is also better than either fixed dropout and data augmentation alone. Learning has been slowed without overfitting, allowing continued improvement.\nThe plot also suggests that learning may not have stalled and may have continued to improve if allowed to continue, but perhaps very modestly.\nResults might be further improved if a pattern of increasing dropout was used instead of a fixed dropout rate throughout the depth of the model.\nLine Plots of Learning Curves for Baseline Model With Dropout and Data Augmentation on the CIFAR-10 Dataset\nDropout and Data Augmentation and Batch Normalization\nWe can expand upon the previous example in a few ways.\nFirst, we can increase the number of training epochs from 200 to 400, to give the model more of an opportunity to improve.\nNext, we can add\nbatch normalization\nin an effort to stabilize the learning and perhaps accelerate the learning process. To offset this acceleration, we can increase the regularization by changing the dropout from a fixed pattern to an increasing pattern.\nThe updated model definition is listed below.\n# define cnn model\ndef define_model():\n\tmodel = Sequential()\n\tmodel.add(Conv2D(32, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same', input_shape=(32, 32, 3)))\n\tmodel.add(BatchNormalization())\n\tmodel.add(Conv2D(32, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))\n\tmodel.add(BatchNormalization())\n\tmodel.add(MaxPooling2D((2, 2)))\n\tmodel.add(Dropout(0.2))\n\tmodel.add(Conv2D(64, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))\n\tmodel.add(BatchNormalization())\n\tmodel.add(Conv2D(64, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))\n\tmodel.add(BatchNormalization())\n\tmodel.add(MaxPooling2D((2, 2)))\n\tmodel.add(Dropout(0.3))\n\tmodel.add(Conv2D(128, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))\n\tmodel.add(BatchNormalization())\n\tmodel.add(Conv2D(128, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))\n\tmodel.add(BatchNormalization())\n\tmodel.add(MaxPooling2D((2, 2)))\n\tmodel.add(Dropout(0.4))\n\tmodel.add(Flatten())\n\tmodel.add(Dense(128, activation='relu', kernel_initializer='he_uniform'))\n\tmodel.add(BatchNormalization())\n\tmodel.add(Dropout(0.5))\n\tmodel.add(Dense(10, activation='softmax'))\n\t# compile model\n\topt = SGD(lr=0.001, momentum=0.9)\n\tmodel.compile(optimizer=opt, loss='categorical_crossentropy', metrics=['accuracy'])\n\treturn model\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n14\n15\n16\n17\n18\n19\n20\n21\n22\n23\n24\n25\n26\n27\n28\n29\n30\n# define cnn model\ndef\ndefine_model\n(\n)\n:\nmodel\n=\nSequential\n(\n)\nmodel\n.\nadd\n(\nConv2D\n(\n32\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n,\ninput_shape\n=\n(\n32\n,\n32\n,\n3\n)\n)\n)\nmodel\n.\nadd\n(\nBatchNormalization\n(\n)\n)\nmodel\n.\nadd\n(\nConv2D\n(\n32\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n)\n)\nmodel\n.\nadd\n(\nBatchNormalization\n(\n)\n)\nmodel\n.\nadd\n(\nMaxPooling2D\n(\n(\n2\n,\n2\n)\n)\n)\nmodel\n.\nadd\n(\nDropout\n(\n0.2\n)\n)\nmodel\n.\nadd\n(\nConv2D\n(\n64\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n)\n)\nmodel\n.\nadd\n(\nBatchNormalization\n(\n)\n)\nmodel\n.\nadd\n(\nConv2D\n(\n64\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n)\n)\nmodel\n.\nadd\n(\nBatchNormalization\n(\n)\n)\nmodel\n.\nadd\n(\nMaxPooling2D\n(\n(\n2\n,\n2\n)\n)\n)\nmodel\n.\nadd\n(\nDropout\n(\n0.3\n)\n)\nmodel\n.\nadd\n(\nConv2D\n(\n128\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n)\n)\nmodel\n.\nadd\n(\nBatchNormalization\n(\n)\n)\nmodel\n.\nadd\n(\nConv2D\n(\n128\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n)\n)\nmodel\n.\nadd\n(\nBatchNormalization\n(\n)\n)\nmodel\n.\nadd\n(\nMaxPooling2D\n(\n(\n2\n,\n2\n)\n)\n)\nmodel\n.\nadd\n(\nDropout\n(\n0.4\n)\n)\nmodel\n.\nadd\n(\nFlatten\n(\n)\n)\nmodel\n.\nadd\n(\nDense\n(\n128\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n)\n)\nmodel\n.\nadd\n(\nBatchNormalization\n(\n)\n)\nmodel\n.\nadd\n(\nDropout\n(\n0.5\n)\n)\nmodel\n.\nadd\n(\nDense\n(\n10\n,\nactivation\n=\n'softmax'\n)\n)\n# compile model\nopt\n=\nSGD\n(\nlr\n=\n0.001\n,\nmomentum\n=\n0.9\n)\nmodel\n.\ncompile\n(\noptimizer\n=\nopt\n,\nloss\n=\n'categorical_crossentropy'\n,\nmetrics\n=\n[\n'accuracy'\n]\n)\nreturn\nmodel\nThe full code listing of a model with increasing dropout, data augmentation, batch normalization, and 400 training epochs is provided below for completeness.\n# baseline model with dropout and data augmentation on the cifar10 dataset\nimport sys\nfrom matplotlib import pyplot\nfrom keras.datasets import cifar10\nfrom keras.utils import to_categorical\nfrom keras.models import Sequential\nfrom keras.layers import Conv2D\nfrom keras.layers import MaxPooling2D\nfrom keras.layers import Dense\nfrom keras.layers import Flatten\nfrom keras.optimizers import SGD\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom keras.layers import Dropout\nfrom keras.layers import BatchNormalization\n\n# load train and test dataset\ndef load_dataset():\n\t# load dataset\n\t(trainX, trainY), (testX, testY) = cifar10.load_data()\n\t# one hot encode target values\n\ttrainY = to_categorical(trainY)\n\ttestY = to_categorical(testY)\n\treturn trainX, trainY, testX, testY\n\n# scale pixels\ndef prep_pixels(train, test):\n\t# convert from integers to floats\n\ttrain_norm = train.astype('float32')\n\ttest_norm = test.astype('float32')\n\t# normalize to range 0-1\n\ttrain_norm = train_norm / 255.0\n\ttest_norm = test_norm / 255.0\n\t# return normalized images\n\treturn train_norm, test_norm\n\n# define cnn model\ndef define_model():\n\tmodel = Sequential()\n\tmodel.add(Conv2D(32, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same', input_shape=(32, 32, 3)))\n\tmodel.add(BatchNormalization())\n\tmodel.add(Conv2D(32, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))\n\tmodel.add(BatchNormalization())\n\tmodel.add(MaxPooling2D((2, 2)))\n\tmodel.add(Dropout(0.2))\n\tmodel.add(Conv2D(64, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))\n\tmodel.add(BatchNormalization())\n\tmodel.add(Conv2D(64, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))\n\tmodel.add(BatchNormalization())\n\tmodel.add(MaxPooling2D((2, 2)))\n\tmodel.add(Dropout(0.3))\n\tmodel.add(Conv2D(128, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))\n\tmodel.add(BatchNormalization())\n\tmodel.add(Conv2D(128, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))\n\tmodel.add(BatchNormalization())\n\tmodel.add(MaxPooling2D((2, 2)))\n\tmodel.add(Dropout(0.4))\n\tmodel.add(Flatten())\n\tmodel.add(Dense(128, activation='relu', kernel_initializer='he_uniform'))\n\tmodel.add(BatchNormalization())\n\tmodel.add(Dropout(0.5))\n\tmodel.add(Dense(10, activation='softmax'))\n\t# compile model\n\topt = SGD(lr=0.001, momentum=0.9)\n\tmodel.compile(optimizer=opt, loss='categorical_crossentropy', metrics=['accuracy'])\n\treturn model\n\n# plot diagnostic learning curves\ndef summarize_diagnostics(history):\n\t# plot loss\n\tpyplot.subplot(211)\n\tpyplot.title('Cross Entropy Loss')\n\tpyplot.plot(history.history['loss'], color='blue', label='train')\n\tpyplot.plot(history.history['val_loss'], color='orange', label='test')\n\t# plot accuracy\n\tpyplot.subplot(212)\n\tpyplot.title('Classification Accuracy')\n\tpyplot.plot(history.history['accuracy'], color='blue', label='train')\n\tpyplot.plot(history.history['val_accuracy'], color='orange', label='test')\n\t# save plot to file\n\tfilename = sys.argv[0].split('/')[-1]\n\tpyplot.savefig(filename + '_plot.png')\n\tpyplot.close()\n\n# run the test harness for evaluating a model\ndef run_test_harness():\n\t# load dataset\n\ttrainX, trainY, testX, testY = load_dataset()\n\t# prepare pixel data\n\ttrainX, testX = prep_pixels(trainX, testX)\n\t# define model\n\tmodel = define_model()\n\t# create data generator\n\tdatagen = ImageDataGenerator(width_shift_range=0.1, height_shift_range=0.1, horizontal_flip=True)\n\t# prepare iterator\n\tit_train = datagen.flow(trainX, trainY, batch_size=64)\n\t# fit model\n\tsteps = int(trainX.shape[0] / 64)\n\thistory = model.fit_generator(it_train, steps_per_epoch=steps, epochs=400, validation_data=(testX, testY), verbose=0)\n\t# evaluate model\n\t_, acc = model.evaluate(testX, testY, verbose=0)\n\tprint('> %.3f' % (acc * 100.0))\n\t# learning curves\n\tsummarize_diagnostics(history)\n\n# entry point, run the test harness\nrun_test_harness()\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n14\n15\n16\n17\n18\n19\n20\n21\n22\n23\n24\n25\n26\n27\n28\n29\n30\n31\n32\n33\n34\n35\n36\n37\n38\n39\n40\n41\n42\n43\n44\n45\n46\n47\n48\n49\n50\n51\n52\n53\n54\n55\n56\n57\n58\n59\n60\n61\n62\n63\n64\n65\n66\n67\n68\n69\n70\n71\n72\n73\n74\n75\n76\n77\n78\n79\n80\n81\n82\n83\n84\n85\n86\n87\n88\n89\n90\n91\n92\n93\n94\n95\n96\n97\n98\n99\n100\n101\n102\n103\n104\n105\n106\n# baseline model with dropout and data augmentation on the cifar10 dataset\nimport\nsys\nfrom\nmatplotlib\nimport\npyplot\nfrom\nkeras\n.\ndatasets\nimport\ncifar10\nfrom\nkeras\n.\nutils\nimport\nto_categorical\nfrom\nkeras\n.\nmodels\nimport\nSequential\nfrom\nkeras\n.\nlayers\nimport\nConv2D\nfrom\nkeras\n.\nlayers\nimport\nMaxPooling2D\nfrom\nkeras\n.\nlayers\nimport\nDense\nfrom\nkeras\n.\nlayers\nimport\nFlatten\nfrom\nkeras\n.\noptimizers\nimport\nSGD\nfrom\nkeras\n.\npreprocessing\n.\nimage\nimport\nImageDataGenerator\nfrom\nkeras\n.\nlayers\nimport\nDropout\nfrom\nkeras\n.\nlayers\nimport\nBatchNormalization\n# load train and test dataset\ndef\nload_dataset\n(\n)\n:\n# load dataset\n(\ntrainX\n,\ntrainY\n)\n,\n(\ntestX\n,\ntestY\n)\n=\ncifar10\n.\nload_data\n(\n)\n# one hot encode target values\ntrainY\n=\nto_categorical\n(\ntrainY\n)\ntestY\n=\nto_categorical\n(\ntestY\n)\nreturn\ntrainX\n,\ntrainY\n,\ntestX\n,\ntestY\n# scale pixels\ndef\nprep_pixels\n(\ntrain\n,\ntest\n)\n:\n# convert from integers to floats\ntrain_norm\n=\ntrain\n.\nastype\n(\n'float32'\n)\ntest_norm\n=\ntest\n.\nastype\n(\n'float32'\n)\n# normalize to range 0-1\ntrain_norm\n=\ntrain_norm\n/\n255.0\ntest_norm\n=\ntest_norm\n/\n255.0\n# return normalized images\nreturn\ntrain_norm\n,\ntest\n_\nnorm\n# define cnn model\ndef\ndefine_model\n(\n)\n:\nmodel\n=\nSequential\n(\n)\nmodel\n.\nadd\n(\nConv2D\n(\n32\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n,\ninput_shape\n=\n(\n32\n,\n32\n,\n3\n)\n)\n)\nmodel\n.\nadd\n(\nBatchNormalization\n(\n)\n)\nmodel\n.\nadd\n(\nConv2D\n(\n32\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n)\n)\nmodel\n.\nadd\n(\nBatchNormalization\n(\n)\n)\nmodel\n.\nadd\n(\nMaxPooling2D\n(\n(\n2\n,\n2\n)\n)\n)\nmodel\n.\nadd\n(\nDropout\n(\n0.2\n)\n)\nmodel\n.\nadd\n(\nConv2D\n(\n64\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n)\n)\nmodel\n.\nadd\n(\nBatchNormalization\n(\n)\n)\nmodel\n.\nadd\n(\nConv2D\n(\n64\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n)\n)\nmodel\n.\nadd\n(\nBatchNormalization\n(\n)\n)\nmodel\n.\nadd\n(\nMaxPooling2D\n(\n(\n2\n,\n2\n)\n)\n)\nmodel\n.\nadd\n(\nDropout\n(\n0.3\n)\n)\nmodel\n.\nadd\n(\nConv2D\n(\n128\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n)\n)\nmodel\n.\nadd\n(\nBatchNormalization\n(\n)\n)\nmodel\n.\nadd\n(\nConv2D\n(\n128\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n)\n)\nmodel\n.\nadd\n(\nBatchNormalization\n(\n)\n)\nmodel\n.\nadd\n(\nMaxPooling2D\n(\n(\n2\n,\n2\n)\n)\n)\nmodel\n.\nadd\n(\nDropout\n(\n0.4\n)\n)\nmodel\n.\nadd\n(\nFlatten\n(\n)\n)\nmodel\n.\nadd\n(\nDense\n(\n128\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n)\n)\nmodel\n.\nadd\n(\nBatchNormalization\n(\n)\n)\nmodel\n.\nadd\n(\nDropout\n(\n0.5\n)\n)\nmodel\n.\nadd\n(\nDense\n(\n10\n,\nactivation\n=\n'softmax'\n)\n)\n# compile model\nopt\n=\nSGD\n(\nlr\n=\n0.001\n,\nmomentum\n=\n0.9\n)\nmodel\n.\ncompile\n(\noptimizer\n=\nopt\n,\nloss\n=\n'categorical_crossentropy'\n,\nmetrics\n=\n[\n'accuracy'\n]\n)\nreturn\nmodel\n# plot diagnostic learning curves\ndef\nsummarize_diagnostics\n(\nhistory\n)\n:\n# plot loss\npyplot\n.\nsubplot\n(\n211\n)\npyplot\n.\ntitle\n(\n'Cross Entropy Loss'\n)\npyplot\n.\nplot\n(\nhistory\n.\nhistory\n[\n'loss'\n]\n,\ncolor\n=\n'blue'\n,\nlabel\n=\n'train'\n)\npyplot\n.\nplot\n(\nhistory\n.\nhistory\n[\n'val_loss'\n]\n,\ncolor\n=\n'orange'\n,\nlabel\n=\n'test'\n)\n# plot accuracy\npyplot\n.\nsubplot\n(\n212\n)\npyplot\n.\ntitle\n(\n'Classification Accuracy'\n)\npyplot\n.\nplot\n(\nhistory\n.\nhistory\n[\n'accuracy'\n]\n,\ncolor\n=\n'blue'\n,\nlabel\n=\n'train'\n)\npyplot\n.\nplot\n(\nhistory\n.\nhistory\n[\n'val_accuracy'\n]\n,\ncolor\n=\n'orange'\n,\nlabel\n=\n'test'\n)\n# save plot to file\nfilename\n=\nsys\n.\nargv\n[\n0\n]\n.\nsplit\n(\n'/'\n)\n[\n-\n1\n]\npyplot\n.\nsavefig\n(\nfilename\n+\n'_plot.png'\n)\npyplot\n.\nclose\n(\n)\n# run the test harness for evaluating a model\ndef\nrun_test_harness\n(\n)\n:\n# load dataset\ntrainX\n,\ntrainY\n,\ntestX\n,\ntestY\n=\nload_dataset\n(\n)\n# prepare pixel data\ntrainX\n,\ntestX\n=\nprep_pixels\n(\ntrainX\n,\ntestX\n)\n# define model\nmodel\n=\ndefine_model\n(\n)\n# create data generator\ndatagen\n=\nImageDataGenerator\n(\nwidth_shift_range\n=\n0.1\n,\nheight_shift_range\n=\n0.1\n,\nhorizontal_flip\n=\nTrue\n)\n# prepare iterator\nit_train\n=\ndatagen\n.\nflow\n(\ntrainX\n,\ntrainY\n,\nbatch_size\n=\n64\n)\n# fit model\nsteps\n=\nint\n(\ntrainX\n.\nshape\n[\n0\n]\n/\n64\n)\nhistory\n=\nmodel\n.\nfit_generator\n(\nit_train\n,\nsteps_per_epoch\n=\nsteps\n,\nepochs\n=\n400\n,\nvalidation_data\n=\n(\ntestX\n,\ntestY\n)\n,\nverbose\n=\n0\n)\n# evaluate model\n_\n,\nacc\n=\nmodel\n.\nevaluate\n(\ntestX\n,\ntestY\n,\nverbose\n=\n0\n)\nprint\n(\n'> %.3f'\n%\n(\nacc *\n100.0\n)\n)\n# learning curves\nsummarize_diagnostics\n(\nhistory\n)\n# entry point, run the test harness\nrun_test_harness\n(\n)\nRunning the model in the test harness prints the classification accuracy on the test dataset.\nNote\n: Your\nresults may vary\ngiven the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.\nIn this case, we can see that we achieved a further lift in model performance to about 88% accuracy, improving upon both dropout and data augmentation alone at about 84% and upon the increasing dropout alone at about 85%.\n> 88.620\n1\n> 88.620\nReviewing the learning curves, we can see the training of the model shows continued improvement for nearly the duration of 400 epochs. We can see perhaps a slight drop-off on the test dataset at around 300 epochs, but the improvement trend does continue.\nThe model may benefit from further training epochs.\nLine Plots of Learning Curves for Baseline Model With Increasing Dropout, Data Augmentation, and Batch Normalization on the CIFAR-10 Dataset\nDiscussion\nIn this section, we explored two approaches designed to expand upon changes to the model that we know already result in an improvement\nA summary of the results is provided below:\nBaseline + Increasing Dropout\n: 84.690%\nBaseline + Dropout + Data Augmentation\n: 85.880%\nBaseline + Increasing Dropout + Data Augmentation + Batch Normalization\n: 88.620%\nThe model is now learning well and we have good control over the rate of learning without overfitting.\nWe might be able to achieve further improvements with additional regularization. This could be achieved with more aggressive dropout in later layers. It is possible that further addition of weight decay may improve the model.\nSo far, we have not tuned the hyperparameters of the learning algorithm, such as the learning rate, which is perhaps the most important hyperparameter. We may expect further improvements with adaptive changes to the learning rate, such as use of an adaptive learning rate technique such as\nAdam\n. These types of changes may help to refine the model once converged.\nHow to Finalize the Model and Make Predictions\nThe process of model improvement may continue for as long as we have ideas and the time and resources to test them out.\nAt some point, a final model configuration must be chosen and adopted. In this case, we will keep things simple and use the baseline model (VGG with 3 blocks) as the final model.\nFirst, we will finalize our model by fitting a model on the entire training dataset and saving the model to file for later use. We will then load the model and evaluate its performance on the hold out test dataset, to get an idea of how well the chosen model actually performs in practice. Finally, we will use the saved model to make a prediction on a single image.\nSave Final Model\nA final model is typically fit on all available data, such as the combination of all train and test dataset.\nIn this tutorial, we will demonstrate the final model fit only on the just training dataset to keep the example simple.\nThe first step is to fit the final model on the entire training dataset.\n# fit model\nmodel.fit(trainX, trainY, epochs=100, batch_size=64, verbose=0)\n1\n2\n# fit model\nmodel\n.\nfit\n(\ntrainX\n,\ntrainY\n,\nepochs\n=\n100\n,\nbatch_size\n=\n64\n,\nverbose\n=\n0\n)\nOnce fit, we can save the final model to an H5 file by calling the\nsave()\nfunction on the model and pass in the chosen filename.\n# save model\nmodel.save('final_model.h5')\n1\n2\n# save model\nmodel\n.\nsave\n(\n'final_model.h5'\n)\nNote: saving and loading a Keras model requires that the\nh5py library\nis installed on your workstation.\nThe complete example of fitting the final model on the training dataset and saving it to file is listed below.\n# save the final model to file\nfrom keras.datasets import cifar10\nfrom keras.utils import to_categorical\nfrom keras.models import Sequential\nfrom keras.layers import Conv2D\nfrom keras.layers import MaxPooling2D\nfrom keras.layers import Dense\nfrom keras.layers import Flatten\nfrom keras.optimizers import SGD\n\n# load train and test dataset\ndef load_dataset():\n\t# load dataset\n\t(trainX, trainY), (testX, testY) = cifar10.load_data()\n\t# one hot encode target values\n\ttrainY = to_categorical(trainY)\n\ttestY = to_categorical(testY)\n\treturn trainX, trainY, testX, testY\n\n# scale pixels\ndef prep_pixels(train, test):\n\t# convert from integers to floats\n\ttrain_norm = train.astype('float32')\n\ttest_norm = test.astype('float32')\n\t# normalize to range 0-1\n\ttrain_norm = train_norm / 255.0\n\ttest_norm = test_norm / 255.0\n\t# return normalized images\n\treturn train_norm, test_norm\n\n# define cnn model\ndef define_model():\n\tmodel = Sequential()\n\tmodel.add(Conv2D(32, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same', input_shape=(32, 32, 3)))\n\tmodel.add(Conv2D(32, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))\n\tmodel.add(MaxPooling2D((2, 2)))\n\tmodel.add(Conv2D(64, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))\n\tmodel.add(Conv2D(64, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))\n\tmodel.add(MaxPooling2D((2, 2)))\n\tmodel.add(Conv2D(128, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))\n\tmodel.add(Conv2D(128, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))\n\tmodel.add(MaxPooling2D((2, 2)))\n\tmodel.add(Flatten())\n\tmodel.add(Dense(128, activation='relu', kernel_initializer='he_uniform'))\n\tmodel.add(Dense(10, activation='softmax'))\n\t# compile model\n\topt = SGD(lr=0.001, momentum=0.9)\n\tmodel.compile(optimizer=opt, loss='categorical_crossentropy', metrics=['accuracy'])\n\treturn model\n\n# run the test harness for evaluating a model\ndef run_test_harness():\n\t# load dataset\n\ttrainX, trainY, testX, testY = load_dataset()\n\t# prepare pixel data\n\ttrainX, testX = prep_pixels(trainX, testX)\n\t# define model\n\tmodel = define_model()\n\t# fit model\n\tmodel.fit(trainX, trainY, epochs=100, batch_size=64, verbose=0)\n\t# save model\n\tmodel.save('final_model.h5')\n\n# entry point, run the test harness\nrun_test_harness()\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n14\n15\n16\n17\n18\n19\n20\n21\n22\n23\n24\n25\n26\n27\n28\n29\n30\n31\n32\n33\n34\n35\n36\n37\n38\n39\n40\n41\n42\n43\n44\n45\n46\n47\n48\n49\n50\n51\n52\n53\n54\n55\n56\n57\n58\n59\n60\n61\n62\n63\n64\n65\n# save the final model to file\nfrom\nkeras\n.\ndatasets\nimport\ncifar10\nfrom\nkeras\n.\nutils\nimport\nto_categorical\nfrom\nkeras\n.\nmodels\nimport\nSequential\nfrom\nkeras\n.\nlayers\nimport\nConv2D\nfrom\nkeras\n.\nlayers\nimport\nMaxPooling2D\nfrom\nkeras\n.\nlayers\nimport\nDense\nfrom\nkeras\n.\nlayers\nimport\nFlatten\nfrom\nkeras\n.\noptimizers\nimport\nSGD\n# load train and test dataset\ndef\nload_dataset\n(\n)\n:\n# load dataset\n(\ntrainX\n,\ntrainY\n)\n,\n(\ntestX\n,\ntestY\n)\n=\ncifar10\n.\nload_data\n(\n)\n# one hot encode target values\ntrainY\n=\nto_categorical\n(\ntrainY\n)\ntestY\n=\nto_categorical\n(\ntestY\n)\nreturn\ntrainX\n,\ntrainY\n,\ntestX\n,\ntestY\n# scale pixels\ndef\nprep_pixels\n(\ntrain\n,\ntest\n)\n:\n# convert from integers to floats\ntrain_norm\n=\ntrain\n.\nastype\n(\n'float32'\n)\ntest_norm\n=\ntest\n.\nastype\n(\n'float32'\n)\n# normalize to range 0-1\ntrain_norm\n=\ntrain_norm\n/\n255.0\ntest_norm\n=\ntest_norm\n/\n255.0\n# return normalized images\nreturn\ntrain_norm\n,\ntest\n_\nnorm\n# define cnn model\ndef\ndefine_model\n(\n)\n:\nmodel\n=\nSequential\n(\n)\nmodel\n.\nadd\n(\nConv2D\n(\n32\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n,\ninput_shape\n=\n(\n32\n,\n32\n,\n3\n)\n)\n)\nmodel\n.\nadd\n(\nConv2D\n(\n32\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n)\n)\nmodel\n.\nadd\n(\nMaxPooling2D\n(\n(\n2\n,\n2\n)\n)\n)\nmodel\n.\nadd\n(\nConv2D\n(\n64\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n)\n)\nmodel\n.\nadd\n(\nConv2D\n(\n64\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n)\n)\nmodel\n.\nadd\n(\nMaxPooling2D\n(\n(\n2\n,\n2\n)\n)\n)\nmodel\n.\nadd\n(\nConv2D\n(\n128\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n)\n)\nmodel\n.\nadd\n(\nConv2D\n(\n128\n,\n(\n3\n,\n3\n)\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n,\npadding\n=\n'same'\n)\n)\nmodel\n.\nadd\n(\nMaxPooling2D\n(\n(\n2\n,\n2\n)\n)\n)\nmodel\n.\nadd\n(\nFlatten\n(\n)\n)\nmodel\n.\nadd\n(\nDense\n(\n128\n,\nactivation\n=\n'relu'\n,\nkernel_initializer\n=\n'he_uniform'\n)\n)\nmodel\n.\nadd\n(\nDense\n(\n10\n,\nactivation\n=\n'softmax'\n)\n)\n# compile model\nopt\n=\nSGD\n(\nlr\n=\n0.001\n,\nmomentum\n=\n0.9\n)\nmodel\n.\ncompile\n(\noptimizer\n=\nopt\n,\nloss\n=\n'categorical_crossentropy'\n,\nmetrics\n=\n[\n'accuracy'\n]\n)\nreturn\nmodel\n# run the test harness for evaluating a model\ndef\nrun_test_harness\n(\n)\n:\n# load dataset\ntrainX\n,\ntrainY\n,\ntestX\n,\ntestY\n=\nload_dataset\n(\n)\n# prepare pixel data\ntrainX\n,\ntestX\n=\nprep_pixels\n(\ntrainX\n,\ntestX\n)\n# define model\nmodel\n=\ndefine_model\n(\n)\n# fit model\nmodel\n.\nfit\n(\ntrainX\n,\ntrainY\n,\nepochs\n=\n100\n,\nbatch_size\n=\n64\n,\nverbose\n=\n0\n)\n# save model\nmodel\n.\nsave\n(\n'final_model.h5'\n)\n# entry point, run the test harness\nrun_test_harness\n(\n)\nAfter running this example you will now have a 4.3-megabyte file with the name ‘\nfinal_model.h5\n‘ in your current working directory.\nEvaluate Final Model\nWe can now load the final model and evaluate it on the hold out test dataset.\nThis is something we might do if we were interested in presenting the performance of the chosen model to project stakeholders.\nThe test dataset was used in the evaluation and choosing among candidate models. As such, it would not make a good final test hold out dataset. Nevertheless, we will use it as a hold out dataset in this case.\nThe model can be loaded via the\nload_model()\nfunction.\nThe complete example of loading the saved model and evaluating it on the test dataset is listed below.\n# evaluate the deep model on the test dataset\nfrom keras.datasets import cifar10\nfrom keras.models import load_model\nfrom keras.utils import to_categorical\n\n# load train and test dataset\ndef load_dataset():\n\t# load dataset\n\t(trainX, trainY), (testX, testY) = cifar10.load_data()\n\t# one hot encode target values\n\ttrainY = to_categorical(trainY)\n\ttestY = to_categorical(testY)\n\treturn trainX, trainY, testX, testY\n\n# scale pixels\ndef prep_pixels(train, test):\n\t# convert from integers to floats\n\ttrain_norm = train.astype('float32')\n\ttest_norm = test.astype('float32')\n\t# normalize to range 0-1\n\ttrain_norm = train_norm / 255.0\n\ttest_norm = test_norm / 255.0\n\t# return normalized images\n\treturn train_norm, test_norm\n\n# run the test harness for evaluating a model\ndef run_test_harness():\n\t# load dataset\n\ttrainX, trainY, testX, testY = load_dataset()\n\t# prepare pixel data\n\ttrainX, testX = prep_pixels(trainX, testX)\n\t# load model\n\tmodel = load_model('final_model.h5')\n\t# evaluate model on test dataset\n\t_, acc = model.evaluate(testX, testY, verbose=0)\n\tprint('> %.3f' % (acc * 100.0))\n\n# entry point, run the test harness\nrun_test_harness()\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n14\n15\n16\n17\n18\n19\n20\n21\n22\n23\n24\n25\n26\n27\n28\n29\n30\n31\n32\n33\n34\n35\n36\n37\n38\n39\n# evaluate the deep model on the test dataset\nfrom\nkeras\n.\ndatasets\nimport\ncifar10\nfrom\nkeras\n.\nmodels\nimport\nload_model\nfrom\nkeras\n.\nutils\nimport\nto\n_\ncategorical\n# load train and test dataset\ndef\nload_dataset\n(\n)\n:\n# load dataset\n(\ntrainX\n,\ntrainY\n)\n,\n(\ntestX\n,\ntestY\n)\n=\ncifar10\n.\nload_data\n(\n)\n# one hot encode target values\ntrainY\n=\nto_categorical\n(\ntrainY\n)\ntestY\n=\nto_categorical\n(\ntestY\n)\nreturn\ntrainX\n,\ntrainY\n,\ntestX\n,\ntestY\n# scale pixels\ndef\nprep_pixels\n(\ntrain\n,\ntest\n)\n:\n# convert from integers to floats\ntrain_norm\n=\ntrain\n.\nastype\n(\n'float32'\n)\ntest_norm\n=\ntest\n.\nastype\n(\n'float32'\n)\n# normalize to range 0-1\ntrain_norm\n=\ntrain_norm\n/\n255.0\ntest_norm\n=\ntest_norm\n/\n255.0\n# return normalized images\nreturn\ntrain_norm\n,\ntest\n_\nnorm\n# run the test harness for evaluating a model\ndef\nrun_test_harness\n(\n)\n:\n# load dataset\ntrainX\n,\ntrainY\n,\ntestX\n,\ntestY\n=\nload_dataset\n(\n)\n# prepare pixel data\ntrainX\n,\ntestX\n=\nprep_pixels\n(\ntrainX\n,\ntestX\n)\n# load model\nmodel\n=\nload_model\n(\n'final_model.h5'\n)\n# evaluate model on test dataset\n_\n,\nacc\n=\nmodel\n.\nevaluate\n(\ntestX\n,\ntestY\n,\nverbose\n=\n0\n)\nprint\n(\n'> %.3f'\n%\n(\nacc *\n100.0\n)\n)\n# entry point, run the test harness\nrun_test_harness\n(\n)\nRunning the example loads the saved model and evaluates the model on the hold out test dataset.\nThe classification accuracy for the model on the test dataset is calculated and printed.\nNote\n: Your\nresults may vary\ngiven the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.\nIn this case, we can see that the model achieved an accuracy of about 73%, very close to what we saw when we evaluated the model as part of our test harness.\n73.750\n1\n73.750\nMake Prediction\nWe can use our saved model to make a prediction on new images.\nThe model assumes that new images are color, they have been segmented so that one image contains one centered object, and the size of the image is square with the size 32×32 pixels.\nBelow is an image extracted from the CIFAR-10 test dataset. You can save it in your current working directory with the filename ‘\nsample_image.png\n‘.\nDeer\nDownload the Deer Image (sample_image.png)\nWe will pretend this is an entirely new and unseen image, prepared in the required way, and see how we might use our saved model to predict the integer that the image represents.\nFor this example, we expect class “\n4\n” for “\nDeer\n“.\nFirst, we can load the image and force it to the size to be 32×32 pixels. The loaded image can then be resized to have a single channel and represent a single sample in a dataset. The\nload_image()\nfunction implements this and will return the loaded image ready for classification.\nImportantly, the pixel values are prepared in the same way as the pixel values were prepared for the training dataset when fitting the final model, in this case, normalized.\n# load and prepare the image\ndef load_image(filename):\n\t# load the image\n\timg = load_img(filename, target_size=(32, 32))\n\t# convert to array\n\timg = img_to_array(img)\n\t# reshape into a single sample with 3 channels\n\timg = img.reshape(1, 32, 32, 3)\n\t# prepare pixel data\n\timg = img.astype('float32')\n\timg = img / 255.0\n\treturn img\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n# load and prepare the image\ndef\nload_image\n(\nfilename\n)\n:\n# load the image\nimg\n=\nload_img\n(\nfilename\n,\ntarget_size\n=\n(\n32\n,\n32\n)\n)\n# convert to array\nimg\n=\nimg_to_array\n(\nimg\n)\n# reshape into a single sample with 3 channels\nimg\n=\nimg\n.\nreshape\n(\n1\n,\n32\n,\n32\n,\n3\n)\n# prepare pixel data\nimg\n=\nimg\n.\nastype\n(\n'float32'\n)\nimg\n=\nimg\n/\n255.0\nreturn\nimg\nNext, we can load the model as in the previous section and call the\npredict_classes()\nfunction to predict the object in the image.\n# predict the class\nresult = model.predict_classes(img)\n1\n2\n# predict the class\nresult\n=\nmodel\n.\npredict_classes\n(\nimg\n)\nThe complete example is listed below.\n# make a prediction for a new image.\nfrom keras.preprocessing.image import load_img\nfrom keras.preprocessing.image import img_to_array\nfrom keras.models import load_model\n\n# load and prepare the image\ndef load_image(filename):\n\t# load the image\n\timg = load_img(filename, target_size=(32, 32))\n\t# convert to array\n\timg = img_to_array(img)\n\t# reshape into a single sample with 3 channels\n\timg = img.reshape(1, 32, 32, 3)\n\t# prepare pixel data\n\timg = img.astype('float32')\n\timg = img / 255.0\n\treturn img\n\n# load an image and predict the class\ndef run_example():\n\t# load the image\n\timg = load_image('sample_image.png')\n\t# load model\n\tmodel = load_model('final_model.h5')\n\t# predict the class\n\tresult = model.predict_classes(img)\n\tprint(result[0])\n\n# entry point, run the example\nrun_example()\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n14\n15\n16\n17\n18\n19\n20\n21\n22\n23\n24\n25\n26\n27\n28\n29\n30\n# make a prediction for a new image.\nfrom\nkeras\n.\npreprocessing\n.\nimage\nimport\nload_img\nfrom\nkeras\n.\npreprocessing\n.\nimage\nimport\nimg_to_array\nfrom\nkeras\n.\nmodels\nimport\nload\n_\nmodel\n# load and prepare the image\ndef\nload_image\n(\nfilename\n)\n:\n# load the image\nimg\n=\nload_img\n(\nfilename\n,\ntarget_size\n=\n(\n32\n,\n32\n)\n)\n# convert to array\nimg\n=\nimg_to_array\n(\nimg\n)\n# reshape into a single sample with 3 channels\nimg\n=\nimg\n.\nreshape\n(\n1\n,\n32\n,\n32\n,\n3\n)\n# prepare pixel data\nimg\n=\nimg\n.\nastype\n(\n'float32'\n)\nimg\n=\nimg\n/\n255.0\nreturn\nimg\n# load an image and predict the class\ndef\nrun_example\n(\n)\n:\n# load the image\nimg\n=\nload_image\n(\n'sample_image.png'\n)\n# load model\nmodel\n=\nload_model\n(\n'final_model.h5'\n)\n# predict the class\nresult\n=\nmodel\n.\npredict_classes\n(\nimg\n)\nprint\n(\nresult\n[\n0\n]\n)\n# entry point, run the example\nrun_example\n(\n)\nRunning the example first loads and prepares the image, loads the model, and then correctly predicts that the loaded image represents a ‘\ndeer\n‘ or class ‘\n4\n‘.\n4\n1\n4\nExtensions\nThis section lists some ideas for extending the tutorial that you may wish to explore.\nPixel Scaling\n. Explore alternate techniques for scaling the pixels, such as centering and standardization, and compare performance.\nLearning Rates\n. Explore alternate learning rates, adaptive learning rates, and learning rate schedules and compare performance.\nTransfer Learning\n. Explore using transfer learning, such as a pre-trained VGG-16 model on this dataset.\nIf you explore any of these extensions, I’d love to know.\nPost your findings in the comments below.\nFurther Reading\nThis section provides more resources on the topic if you are looking to go deeper.\nPosts\nObject Recognition with Convolutional Neural Networks in the Keras Deep Learning Library\nA Gentle Introduction to Dropout for Regularizing Deep Neural Networks\nUse Weight Regularization to Reduce Overfitting of Deep Learning Models\nAPI\nKeras Datasets API\nKeras Datasets Code\nArticles\nClassification datasets results, What is the class of this image?\nCIFAR-10, Wikipedia\n.\nThe CIFAR-10 dataset and CIFAR-100 datasets\n.\nCIFAR-10 – Object Recognition in Images, Kaggle\n.\nSimple CNN 90%+, Pasquale Giovenale, Kaggle\n.\nSummary\nIn this tutorial, you discovered how to develop a convolutional neural network model from scratch for object photo classification.\nSpecifically, you learned:\nHow to develop a test harness to develop a robust evaluation of a model and establish a baseline of performance for a classification task.\nHow to explore extensions to a baseline model to improve learning and model capacity.\nHow to develop a finalized model, evaluate the performance of the final model, and use it to make predictions on new images.\nDo you have any questions?\nAsk your questions in the comments below and I will do my best to answer.\nDevelop Deep Learning Models for Vision Today!\nDevelop Your Own Vision Models in Minutes\n...with just a few lines of python code\nDiscover how in my new Ebook:\nDeep Learning for Computer Vision\nIt provides\nself-study tutorials\non topics like:\nclassification\n,\nobject detection (yolo and rcnn)\n,\nface recognition (vggface and facenet)\n,\ndata preparation\nand much more...\nFinally Bring Deep Learning to your Vision Projects\nSkip the Academics. Just Results.\nSee What's Inside\nShare\nPost\nShare\nMore On This Topic\nHow to Develop a GAN to Generate CIFAR10 Small Color…\nHow to Develop a Deep Learning Photo Caption…\nHow to Develop a CNN for MNIST Handwritten Digit…\nHow to Develop a Multichannel CNN Model for Text…\nHow to Prepare a Photo Caption Dataset for Training…\nDeep Learning CNN for Fashion-MNIST Clothing Classification\nAbout Jason Brownlee\nJason Brownlee, PhD is a machine learning specialist who teaches developers how to get results with modern machine learning methods via hands-on tutorials.\nView all posts by Jason Brownlee\n→",
  "metadata": {
    "title": "How to Develop a CNN From Scratch for CIFAR-10 Photo Classification - MachineLearningMastery.com",
    "description": null,
    "keywords": null,
    "author": "Jason Brownlee",
    "og:locale": "en_US",
    "og:type": "article",
    "og:title": "How to Develop a CNN From Scratch for CIFAR-10 Photo Classification - MachineLearningMastery.com",
    "og:description": "Discover how to develop a deep convolutional neural network model from scratch for the CIFAR-10 object classification dataset. The CIFAR-10 small photo classification problem is a standard dataset used in computer vision and deep learning. Although the dataset is effectively solved, it can be used as the basis for learning and practicing how to develop, evaluate, and use convolutional deep…",
    "og:url": "https://machinelearningmastery.com/how-to-develop-a-cnn-from-scratch-for-cifar-10-photo-classification/",
    "og:site_name": "MachineLearningMastery.com",
    "og:image": "https://machinelearningmastery.com/wp-content/uploads/2019/02/Line-Plots-of-Learning-Curves-for-Baseline-Model-With-Increasing-Dropout-Data-Augmentation-and-Batch-Normalization-on-the-CIFAR-10-Dataset.png",
    "og:image:width": "640",
    "og:image:height": "480",
    "og:image:type": "image/png",
    "twitter:label1": "Written by",
    "twitter:data1": "Jason Brownlee",
    "twitter:label2": "Est. reading time",
    "twitter:data2": "55 minutes",
    "twitter:card": "summary_large_image",
    "twitter:description": "Discover how to develop a deep convolutional neural network model from scratch for the CIFAR-10 object classification dataset. The CIFAR-10 small photo classification problem is a standard dataset used in computer vision and deep learning. Although the dataset is effectively solved, it can be used as the basis for learning and practicing how to develop, evaluate, and use convolutional deep…",
    "twitter:title": "How to Develop a CNN From Scratch for CIFAR-10 Photo Classification - MachineLearningMastery.com",
    "article:publisher": "https://www.facebook.com/MachineLearningMastery/",
    "article:author": "https://www.facebook.com/MachineLearningMastery/",
    "article:published_time": "2019-05-12T19:00:02+00:00",
    "article:modified_time": "2020-08-27T22:46:37+00:00"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [],
  "images": [
    {
      "status": "ok",
      "url": "https://machinelearningmastery.com/wp-content/uploads/2019/05/How-to-Develop-a-Convolutional-Neural-Network-From-Scratch-for-CIFAR-10-Photo-Classification.jpg",
      "bytes": 39777,
      "reused": true,
      "local_path": "attachments/6dda3c1e3103/images/000.jpg",
      "src": "https://machinelearningmastery.com/wp-content/uploads/2019/05/How-to-Develop-a-Convolutional-Neural-Network-From-Scratch-for-CIFAR-10-Photo-Classification.jpg",
      "alt": "How to Develop a Convolutional Neural Network From Scratch for CIFAR-10 Photo Classification",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://machinelearningmastery.com/wp-content/uploads/2019/01/Plot-of-a-Subset-of-Images-from-the-CIFAR-10-Dataset.png",
      "bytes": 54312,
      "reused": true,
      "local_path": "attachments/6dda3c1e3103/images/002.png",
      "src": "https://machinelearningmastery.com/wp-content/uploads/2019/01/Plot-of-a-Subset-of-Images-from-the-CIFAR-10-Dataset.png",
      "alt": "Plot of a Subset of Images From the CIFAR-10 Dataset",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://machinelearningmastery.com/wp-content/uploads/2019/02/Line-Plots-of-Learning-Curves-for-VGG-1-Baseline-on-the-CIFAR-10-Dataset.png",
      "bytes": 7032,
      "reused": true,
      "local_path": "attachments/6dda3c1e3103/images/006.png",
      "src": "https://machinelearningmastery.com/wp-content/uploads/2019/02/Line-Plots-of-Learning-Curves-for-VGG-1-Baseline-on-the-CIFAR-10-Dataset.png",
      "alt": "Line Plots of Learning Curves for VGG 1 Baseline on the CIFAR-10 Dataset",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://machinelearningmastery.com/wp-content/uploads/2019/02/Line-Plots-of-Learning-Curves-for-VGG-2-Baseline-on-the-CIFAR-10-Dataset.png",
      "bytes": 7440,
      "reused": true,
      "local_path": "attachments/6dda3c1e3103/images/008.png",
      "src": "https://machinelearningmastery.com/wp-content/uploads/2019/02/Line-Plots-of-Learning-Curves-for-VGG-2-Baseline-on-the-CIFAR-10-Dataset.png",
      "alt": "Line Plots of Learning Curves for VGG 2 Baseline on the CIFAR-10 Dataset",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://machinelearningmastery.com/wp-content/uploads/2019/02/Line-Plots-of-Learning-Curves-for-VGG-3-Baseline-on-the-CIFAR-10-Dataset.png",
      "bytes": 7463,
      "reused": true,
      "local_path": "attachments/6dda3c1e3103/images/010.png",
      "src": "https://machinelearningmastery.com/wp-content/uploads/2019/02/Line-Plots-of-Learning-Curves-for-VGG-3-Baseline-on-the-CIFAR-10-Dataset.png",
      "alt": "Line Plots of Learning Curves for VGG 3 Baseline on the CIFAR-10 Dataset",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://machinelearningmastery.com/wp-content/uploads/2019/02/Line-Plots-of-Learning-Curves-for-Baseline-Model-With-Dropout-on-the-CIFAR-10-Dataset.png",
      "bytes": 8614,
      "reused": true,
      "local_path": "attachments/6dda3c1e3103/images/012.png",
      "src": "https://machinelearningmastery.com/wp-content/uploads/2019/02/Line-Plots-of-Learning-Curves-for-Baseline-Model-With-Dropout-on-the-CIFAR-10-Dataset.png",
      "alt": "Line Plots of Learning Curves for Baseline Model With Dropout on the CIFAR-10 Dataset",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://machinelearningmastery.com/wp-content/uploads/2019/02/Line-Plots-of-Learning-Curves-for-Baseline-Model-With-Weight-Decay-on-the-CIFAR-10-Dataset.png",
      "bytes": 9173,
      "reused": true,
      "local_path": "attachments/6dda3c1e3103/images/014.png",
      "src": "https://machinelearningmastery.com/wp-content/uploads/2019/02/Line-Plots-of-Learning-Curves-for-Baseline-Model-With-Weight-Decay-on-the-CIFAR-10-Dataset.png",
      "alt": "Line Plots of Learning Curves for Baseline Model With Weight Decay on the CIFAR-10 Dataset",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://machinelearningmastery.com/wp-content/uploads/2019/02/Line-Plots-of-Learning-Curves-for-Baseline-Model-With-Data-Augmentation-on-the-CIFAR-10-Dataset.png",
      "bytes": 9253,
      "reused": true,
      "local_path": "attachments/6dda3c1e3103/images/016.png",
      "src": "https://machinelearningmastery.com/wp-content/uploads/2019/02/Line-Plots-of-Learning-Curves-for-Baseline-Model-With-Data-Augmentation-on-the-CIFAR-10-Dataset.png",
      "alt": "Line Plots of Learning Curves for Baseline Model With Data Augmentation on the CIFAR-10 Dataset",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://machinelearningmastery.com/wp-content/uploads/2019/02/Line-Plots-of-Learning-Curves-for-Baseline-Model-With-Increasing-Dropout-on-the-CIFAR-10-Dataset.png",
      "bytes": 9721,
      "reused": true,
      "local_path": "attachments/6dda3c1e3103/images/018.png",
      "src": "https://machinelearningmastery.com/wp-content/uploads/2019/02/Line-Plots-of-Learning-Curves-for-Baseline-Model-With-Increasing-Dropout-on-the-CIFAR-10-Dataset.png",
      "alt": "Line Plots of Learning Curves for Baseline Model With Increasing Dropout on the CIFAR-10 Dataset",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://machinelearningmastery.com/wp-content/uploads/2019/02/Line-Plots-of-Learning-Curves-for-Baseline-Model-With-Dropout-and-Data-Augmentation-on-the-CIFAR-10-Dataset.png",
      "bytes": 10276,
      "reused": true,
      "local_path": "attachments/6dda3c1e3103/images/020.png",
      "src": "https://machinelearningmastery.com/wp-content/uploads/2019/02/Line-Plots-of-Learning-Curves-for-Baseline-Model-With-Dropout-and-Data-Augmentation-on-the-CIFAR-10-Dataset.png",
      "alt": "Line Plots of Learning Curves for Baseline Model With Dropout and Data Augmentation on the CIFAR-10 Dataset",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://machinelearningmastery.com/wp-content/uploads/2019/02/Line-Plots-of-Learning-Curves-for-Baseline-Model-With-Increasing-Dropout-Data-Augmentation-and-Batch-Normalization-on-the-CIFAR-10-Dataset.png",
      "bytes": 9634,
      "reused": true,
      "local_path": "attachments/6dda3c1e3103/images/022.png",
      "src": "https://machinelearningmastery.com/wp-content/uploads/2019/02/Line-Plots-of-Learning-Curves-for-Baseline-Model-With-Increasing-Dropout-Data-Augmentation-and-Batch-Normalization-on-the-CIFAR-10-Dataset.png",
      "alt": "Line Plots of Learning Curves for Baseline Model With Increasing Dropout, Data Augmentation, and Batch Normalization on the CIFAR-10 Dataset",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://machinelearningmastery.com/wp-content/uploads/2019/02/sample_image-1-300x298.png",
      "bytes": 17227,
      "reused": true,
      "local_path": "attachments/6dda3c1e3103/images/024.png",
      "src": "https://machinelearningmastery.com/wp-content/uploads/2019/02/sample_image-1-300x298.png",
      "alt": "Deer",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://machinelearningmastery.com/wp-content/uploads/2022/11/DL4CV-220.png",
      "bytes": 7552,
      "reused": false,
      "local_path": "attachments/6dda3c1e3103/images/body-8bdaaeee1207cd36fc6e.png",
      "src": "https://machinelearningmastery.com/wp-content/uploads/2022/11/DL4CV-220.png",
      "alt": "Deep Learning for Computer Vision",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://machinelearningmastery.com/wp-content/uploads/2019/05/Example-of-100-GAN-Generated-CIFAR-10-Small-Object-Photographs.png",
      "bytes": 341898,
      "reused": true,
      "local_path": "attachments/6dda3c1e3103/images/028.png",
      "src": "https://machinelearningmastery.com/wp-content/uploads/2019/05/Example-of-100-GAN-Generated-CIFAR-10-Small-Object-Photographs.png",
      "alt": "Example of 100 GAN Generated CIFAR-10 Small Object Photographs",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://machinelearningmastery.com/wp-content/uploads/2017/09/example.jpg",
      "bytes": 60698,
      "reused": true,
      "local_path": "attachments/6dda3c1e3103/images/032.jpg",
      "src": "https://machinelearningmastery.com/wp-content/uploads/2017/09/example.jpg",
      "alt": "Photo of a dog at the beach.",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://machinelearningmastery.com/wp-content/uploads/2019/02/Plot-of-a-Subset-of-Images-from-the-MNIST-Dataset.png",
      "bytes": 12626,
      "reused": true,
      "local_path": "attachments/6dda3c1e3103/images/034.png",
      "src": "https://machinelearningmastery.com/wp-content/uploads/2019/02/Plot-of-a-Subset-of-Images-from-the-MNIST-Dataset.png",
      "alt": "Plot of a Subset of Images From the MNIST Dataset",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://machinelearningmastery.com/wp-content/uploads/2017/10/Plot-of-the-Multichannel-Convolutional-Neural-Network-For-Text.png",
      "bytes": 38747,
      "reused": true,
      "local_path": "attachments/6dda3c1e3103/images/038.png",
      "src": "https://machinelearningmastery.com/wp-content/uploads/2017/10/Plot-of-the-Multichannel-Convolutional-Neural-Network-For-Text.png",
      "alt": "Plot of the Multichannel Convolutional Neural Network For Text",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://machinelearningmastery.com/wp-content/uploads/2017/11/How-to-Prepare-a-Photo-Caption-Dataset-for-Training-a-Deep-Learning-Model.jpg",
      "bytes": 76709,
      "reused": false,
      "local_path": "attachments/6dda3c1e3103/images/body-8ff060dd238d984201fa.jpg",
      "src": "https://machinelearningmastery.com/wp-content/uploads/2017/11/How-to-Prepare-a-Photo-Caption-Dataset-for-Training-a-Deep-Learning-Model.jpg",
      "alt": "How to Prepare a Photo Caption Dataset for Training a Deep Learning Model",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://machinelearningmastery.com/wp-content/uploads/2019/05/Loss-and-Accuracy-Learning-Curves-for-the-Baseline-Model-on-the-Fashion-MNIST-Dataset-During-k-Fold-Cross-Validation.png",
      "bytes": 24458,
      "reused": false,
      "local_path": "attachments/6dda3c1e3103/images/body-c6f9de8f9d5672b0aad0.png",
      "src": "https://machinelearningmastery.com/wp-content/uploads/2019/05/Loss-and-Accuracy-Learning-Curves-for-the-Baseline-Model-on-the-Fashion-MNIST-Dataset-During-k-Fold-Cross-Validation.png",
      "alt": "Loss-and-Accuracy-Learning-Curves-for-the-Baseline-Model-on-the-Fashion-MNIST-Dataset-During-k-Fold-Cross-Validation",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://machinelearningmastery.com/author/jasonb/",
      "label": "Jason Brownlee",
      "type": "a"
    },
    {
      "href": "https://machinelearningmastery.com/category/deep-learning-for-computer-vision/",
      "label": "Deep Learning for Computer Vision",
      "type": "a"
    },
    {
      "href": "https://machinelearningmastery.com/how-to-develop-a-cnn-from-scratch-for-cifar-10-photo-classification/",
      "label": "120",
      "type": "a"
    },
    {
      "href": "https://machinelearningmastery.com/deep-learning-for-computer-vision/",
      "label": "Deep Learning for Computer Vision",
      "type": "a"
    },
    {
      "href": "https://www.flickr.com/photos/rmdobservations/7886626362/",
      "label": "Rose Dlhopolsky",
      "type": "a"
    },
    {
      "href": "https://www.cs.toronto.edu/~kriz/cifar.html",
      "label": "Canadian Institute For Advanced Research",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/CIFAR-10",
      "label": "CIFAR-10 dataset",
      "type": "a"
    },
    {
      "href": "https://machinelearningmastery.com/why-one-hot-encode-data-in-machine-learning/",
      "label": "one hot encoding",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1409.1556",
      "label": "Very Deep Convolutional Networks for Large-Scale Image Recognition",
      "type": "a"
    },
    {
      "href": "https://machinelearningmastery.com/rectified-linear-activation-function-for-deep-learning-neural-networks/",
      "label": "ReLU activation function",
      "type": "a"
    },
    {
      "href": "https://machinelearningmastery.com/how-to-choose-loss-functions-when-training-deep-learning-neural-networks/",
      "label": "categorical cross entropy loss function",
      "type": "a"
    },
    {
      "href": "https://machinelearningmastery.com/different-results-each-time-in-machine-learning/",
      "label": "results may vary",
      "type": "a"
    },
    {
      "href": "https://machinelearningmastery.com/dropout-for-regularizing-deep-neural-networks/",
      "label": "A Gentle Introduction to Dropout for Regularizing Deep Neural Networks",
      "type": "a"
    },
    {
      "href": "https://machinelearningmastery.com/weight-regularization-to-reduce-overfitting-of-deep-learning-models/",
      "label": "Use Weight Regularization to Reduce Overfitting of Deep Learning Models",
      "type": "a"
    },
    {
      "href": "https://keras.io/preprocessing/image/",
      "label": "ImageDataGenerator class",
      "type": "a"
    },
    {
      "href": "https://machinelearningmastery.com/how-to-accelerate-learning-of-deep-neural-networks-with-batch-normalization/",
      "label": "batch normalization",
      "type": "a"
    },
    {
      "href": "https://machinelearningmastery.com/adam-optimization-algorithm-for-deep-learning/",
      "label": "Adam",
      "type": "a"
    },
    {
      "href": "https://www.h5py.org/",
      "label": "h5py library",
      "type": "a"
    },
    {
      "href": "https://machinelearningmastery.com/wp-content/uploads/2019/02/sample_image-1.png",
      "label": "Download the Deer Image (sample_image.png)",
      "type": "a"
    },
    {
      "href": "https://machinelearningmastery.com/object-recognition-convolutional-neural-networks-keras-deep-learning-library/",
      "label": "Object Recognition with Convolutional Neural Networks in the Keras Deep Learning Library",
      "type": "a"
    },
    {
      "href": "https://keras.io/datasets/",
      "label": "Keras Datasets API",
      "type": "a"
    },
    {
      "href": "https://github.com/keras-team/keras/tree/master/keras/datasets",
      "label": "Keras Datasets Code",
      "type": "a"
    },
    {
      "href": "http://rodrigob.github.io/are_we_there_yet/build/classification_datasets_results.html",
      "label": "Classification datasets results, What is the class of this image?",
      "type": "a"
    },
    {
      "href": "https://www.kaggle.com/c/cifar-10",
      "label": "CIFAR-10 – Object Recognition in Images, Kaggle",
      "type": "a"
    },
    {
      "href": "https://www.kaggle.com/c/cifar-10/discussion/40237",
      "label": "Simple CNN 90%+, Pasquale Giovenale, Kaggle",
      "type": "a"
    },
    {
      "href": "https://machinelearningmastery.com/how-to-develop-a-generative-adversarial-network-for-a-cifar-10-small-object-photographs-from-scratch/",
      "label": "How to Develop a GAN to Generate CIFAR10 Small Color…",
      "type": "a"
    },
    {
      "href": "https://machinelearningmastery.com/develop-a-deep-learning-caption-generation-model-in-python/",
      "label": "How to Develop a Deep Learning Photo Caption…",
      "type": "a"
    },
    {
      "href": "https://machinelearningmastery.com/how-to-develop-a-convolutional-neural-network-from-scratch-for-mnist-handwritten-digit-classification/",
      "label": "How to Develop a CNN for MNIST Handwritten Digit…",
      "type": "a"
    },
    {
      "href": "https://machinelearningmastery.com/develop-n-gram-multichannel-convolutional-neural-network-sentiment-analysis/",
      "label": "How to Develop a Multichannel CNN Model for Text…",
      "type": "a"
    },
    {
      "href": "https://machinelearningmastery.com/prepare-photo-caption-dataset-training-deep-learning-model/",
      "label": "How to Prepare a Photo Caption Dataset for Training…",
      "type": "a"
    },
    {
      "href": "https://machinelearningmastery.com/how-to-develop-a-cnn-from-scratch-for-fashion-mnist-clothing-classification/",
      "label": "Deep Learning CNN for Fashion-MNIST Clothing Classification",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:54:20Z",
  "source_url": "https://machinelearningmastery.com/how-to-develop-a-cnn-from-scratch-for-cifar-10-photo-classification/",
  "papers": [
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1409.1556",
      "bytes": 200010,
      "reused": true,
      "local_path": "papers/repair/01560c40deef0c339e01.pdf"
    }
  ],
  "actual_video_files": [],
  "body_chars": 128098,
  "expected_body_images": 19,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/6dda3c1e3103/pages/source.html",
  "local_html": "attachments/6dda3c1e3103/pages/article.html",
  "body_base_path": "attachments/6dda3c1e3103/pages/",
  "body_markdown_path": "attachments/6dda3c1e3103/pages/article.md"
}