{
  "source_file": "0930",
  "source_platform": "技术平台",
  "author": "Jason Brownlee / Machine Learning Mastery",
  "title_listed": "How to Develop LSTM Models for Time Series Forecasting",
  "keywords": "LSTM,时间序列,预测",
  "domain": "机器学习",
  "content_type": "教程",
  "is_paid_flag": "否",
  "url": "https://machinelearningmastery.com/how-to-develop-lstm-models-for-time-series-forecasting/",
  "publish_date": "2019-05",
  "corpus_stage": "预训练/SFT",
  "notes": "LSTM时间序列预测教程",
  "id": "e9465448bb83",
  "collected_at": "2026-10-02T17:54:46",
  "title": "How to Develop LSTM Models for Time Series Forecasting - MachineLearningMastery.com",
  "body_markdown": "How to Develop LSTM Models for Time Series Forecasting\nBy\nJason Brownlee\non\nAugust 28, 2020\nin\nDeep Learning for Time Series\n991\nShare\nPost\nShare\nLong Short-Term Memory networks, or\nLSTMs\nfor short, can be applied to\ntime series forecasting\n.\nThere are many types of LSTM models that can be used for each specific type of time series forecasting problem.\nIn this tutorial, you will discover how to develop a suite of LSTM models for a range of standard\ntime series forecasting problems\n.\nThe objective of this tutorial is to provide standalone examples of each model on each type of time series problem as a template that you can copy and adapt for your specific time series forecasting problem.\nAfter completing this tutorial, you will know:\nHow to develop LSTM models for univariate time series forecasting.\nHow to develop LSTM models for multivariate time series forecasting.\nHow to develop LSTM models for multi-step time series forecasting.\nThis is a large and important post; you may want to bookmark it for future reference.\nKick-start your project\nwith my new book\nDeep Learning for Time Series Forecasting\n, including\nstep-by-step tutorials\nand the\nPython source code\nfiles for all examples.\nLet’s get started.\nHow to Develop LSTM Models for Time Series Forecasting\nPhoto by\nN i c o l a\n, some rights reserved.\nTutorial Overview\nIn this tutorial, we will explore how to develop a suite of different types of LSTM models for time series forecasting.\nThe models are demonstrated on small contrived time series problems intended to give the flavor of the type of time series problem being addressed. The chosen configuration of the models is arbitrary and not optimized for each problem; that was not the goal.\nThis tutorial is divided into four parts; they are:\nUnivariate LSTM Models\nData Preparation\nVanilla LSTM\nStacked LSTM\nBidirectional LSTM\nCNN LSTM\nConvLSTM\nMultivariate LSTM Models\nMultiple Input Series.\nMultiple Parallel Series.\nMulti-Step LSTM Models\nData Preparation\nVector Output Model\nEncoder-Decoder Model\nMultivariate Multi-Step LSTM Models\nMultiple Input Multi-Step Output.\nMultiple Parallel Input and Multi-Step Output.\nUnivariate LSTM Models\nLSTMs can be used to model univariate time series forecasting problems.\nThese are problems comprised of a single series of observations and a model is required to learn from the series of past observations to predict the next value in the sequence.\nWe will demonstrate a number of variations of the LSTM model for univariate time series forecasting.\nThis section is divided into six parts; they are:\nData Preparation\nVanilla LSTM\nStacked LSTM\nBidirectional LSTM\nCNN LSTM\nConvLSTM\nEach of these models are demonstrated for one-step univariate time series forecasting, but can easily be adapted and used as the input part of a model for other types of time series forecasting problems.\nData Preparation\nBefore a univariate series can be modeled, it must be prepared.\nThe LSTM model will learn a function that maps a sequence of past observations as input to an output observation. As such, the sequence of observations must be transformed into multiple examples from which the LSTM can learn.\nConsider a given univariate sequence:\n[10, 20, 30, 40, 50, 60, 70, 80, 90]\n1\n[10, 20, 30, 40, 50, 60, 70, 80, 90]\nWe can divide the sequence into multiple input/output patterns called samples, where three time steps are used as input and one time step is used as output for the one-step prediction that is being learned.\nX,\t\t\t\ty\n10, 20, 30\t\t40\n20, 30, 40\t\t50\n30, 40, 50\t\t60\n...\n1\n2\n3\n4\n5\nX,\t\t\t\ty\n10, 20, 30\t\t40\n20, 30, 40\t\t50\n30, 40, 50\t\t60\n...\nThe\nsplit_sequence()\nfunction below implements this behavior and will split a given univariate sequence into multiple samples where each sample has a specified number of time steps and the output is a single time step.\n# split a univariate sequence into samples\ndef split_sequence(sequence, n_steps):\n\tX, y = list(), list()\n\tfor i in range(len(sequence)):\n\t\t# find the end of this pattern\n\t\tend_ix = i + n_steps\n\t\t# check if we are beyond the sequence\n\t\tif end_ix > len(sequence)-1:\n\t\t\tbreak\n\t\t# gather input and output parts of the pattern\n\t\tseq_x, seq_y = sequence[i:end_ix], sequence[end_ix]\n\t\tX.append(seq_x)\n\t\ty.append(seq_y)\n\treturn array(X), array(y)\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n14\n# split a univariate sequence into samples\ndef\nsplit_sequence\n(\nsequence\n,\nn_steps\n)\n:\nX\n,\ny\n=\nlist\n(\n)\n,\nlist\n(\n)\nfor\ni\nin\nrange\n(\nlen\n(\nsequence\n)\n)\n:\n# find the end of this pattern\nend_ix\n=\ni\n+\nn\n_\nsteps\n# check if we are beyond the sequence\nif\nend_ix\n>\nlen\n(\nsequence\n)\n-\n1\n:\nbreak\n# gather input and output parts of the pattern\nseq_x\n,\nseq_y\n=\nsequence\n[\ni\n:\nend_ix\n]\n,\nsequence\n[\nend_ix\n]\nX\n.\nappend\n(\nseq_x\n)\ny\n.\nappend\n(\nseq_y\n)\nreturn\narray\n(\nX\n)\n,\narray\n(\ny\n)\nWe can demonstrate this function on our small contrived dataset above.\nThe complete example is listed below.\n# univariate data preparation\nfrom numpy import array\n\n# split a univariate sequence into samples\ndef split_sequence(sequence, n_steps):\n\tX, y = list(), list()\n\tfor i in range(len(sequence)):\n\t\t# find the end of this pattern\n\t\tend_ix = i + n_steps\n\t\t# check if we are beyond the sequence\n\t\tif end_ix > len(sequence)-1:\n\t\t\tbreak\n\t\t# gather input and output parts of the pattern\n\t\tseq_x, seq_y = sequence[i:end_ix], sequence[end_ix]\n\t\tX.append(seq_x)\n\t\ty.append(seq_y)\n\treturn array(X), array(y)\n\n# define input sequence\nraw_seq = [10, 20, 30, 40, 50, 60, 70, 80, 90]\n# choose a number of time steps\nn_steps = 3\n# split into samples\nX, y = split_sequence(raw_seq, n_steps)\n# summarize the data\nfor i in range(len(X)):\n\tprint(X[i], y[i])\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n14\n15\n16\n17\n18\n19\n20\n21\n22\n23\n24\n25\n26\n27\n# univariate data preparation\nfrom\nnumpy\nimport\narray\n# split a univariate sequence into samples\ndef\nsplit_sequence\n(\nsequence\n,\nn_steps\n)\n:\nX\n,\ny\n=\nlist\n(\n)\n,\nlist\n(\n)\nfor\ni\nin\nrange\n(\nlen\n(\nsequence\n)\n)\n:\n# find the end of this pattern\nend_ix\n=\ni\n+\nn\n_\nsteps\n# check if we are beyond the sequence\nif\nend_ix\n>\nlen\n(\nsequence\n)\n-\n1\n:\nbreak\n# gather input and output parts of the pattern\nseq_x\n,\nseq_y\n=\nsequence\n[\ni\n:\nend_ix\n]\n,\nsequence\n[\nend_ix\n]\nX\n.\nappend\n(\nseq_x\n)\ny\n.\nappend\n(\nseq_y\n)\nreturn\narray\n(\nX\n)\n,\narray\n(\ny\n)\n# define input sequence\nraw_seq\n=\n[\n10\n,\n20\n,\n30\n,\n40\n,\n50\n,\n60\n,\n70\n,\n80\n,\n90\n]\n# choose a number of time steps\nn_steps\n=\n3\n# split into samples\nX\n,\ny\n=\nsplit_sequence\n(\nraw_seq\n,\nn_steps\n)\n# summarize the data\nfor\ni\nin\nrange\n(\nlen\n(\nX\n)\n)\n:\nprint\n(\nX\n[\ni\n]\n,\ny\n[\ni\n]\n)\nRunning the example splits the univariate series into six samples where each sample has three input time steps and one output time step.\n[10 20 30] 40\n[20 30 40] 50\n[30 40 50] 60\n[40 50 60] 70\n[50 60 70] 80\n[60 70 80] 90\n1\n2\n3\n4\n5\n6\n[10 20 30] 40\n[20 30 40] 50\n[30 40 50] 60\n[40 50 60] 70\n[50 60 70] 80\n[60 70 80] 90\nNow that we know how to prepare a univariate series for modeling, let’s look at developing LSTM models that can learn the mapping of inputs to outputs, starting with a Vanilla LSTM.\nNeed help with Deep Learning for Time Series?\nTake my free 7-day email crash course now (with sample code).\nClick to sign-up and also get a free PDF Ebook version of the course.\nDownload Your FREE Mini-Course\nVanilla LSTM\nA Vanilla LSTM is an LSTM model that has a single hidden layer of LSTM units, and an output layer used to make a prediction.\nWe can define a Vanilla LSTM for univariate time series forecasting as follows.\n...\n# define model\nmodel = Sequential()\nmodel.add(LSTM(50, activation='relu', input_shape=(n_steps, n_features)))\nmodel.add(Dense(1))\nmodel.compile(optimizer='adam', loss='mse')\n1\n2\n3\n4\n5\n6\n.\n.\n.\n# define model\nmodel\n=\nSequential\n(\n)\nmodel\n.\nadd\n(\nLSTM\n(\n50\n,\nactivation\n=\n'relu'\n,\ninput_shape\n=\n(\nn_steps\n,\nn_features\n)\n)\n)\nmodel\n.\nadd\n(\nDense\n(\n1\n)\n)\nmodel\n.\ncompile\n(\noptimizer\n=\n'adam'\n,\nloss\n=\n'mse'\n)\nKey in the definition is the shape of the input; that is what the model expects as input for each sample in terms of the number of time steps and the number of features.\nWe are working with a univariate series, so the number of features is one, for one variable.\nThe number of time steps as input is the number we chose when preparing our dataset as an argument to the\nsplit_sequence()\nfunction.\nThe shape of the input for each sample is specified in the\ninput_shape\nargument on the definition of first hidden layer.\nWe almost always have multiple samples, therefore, the model will expect the input component of training data to have the dimensions or shape:\n[samples, timesteps, features]\n1\n[samples, timesteps, features]\nOur\nsplit_sequence()\nfunction in the previous section outputs the X with the shape [\nsamples, timesteps\n], so we easily reshape it to have an additional dimension for the one feature.\n...\n# reshape from [samples, timesteps] into [samples, timesteps, features]\nn_features = 1\nX = X.reshape((X.shape[0], X.shape[1], n_features))\n1\n2\n3\n4\n.\n.\n.\n# reshape from [samples, timesteps] into [samples, timesteps, features]\nn_features\n=\n1\nX\n=\nX\n.\nreshape\n(\n(\nX\n.\nshape\n[\n0\n]\n,\nX\n.\nshape\n[\n1\n]\n,\nn_features\n)\n)\nIn this case, we define a model with 50 LSTM units in the hidden layer and an output layer that predicts a single numerical value.\nThe model is fit using the efficient\nAdam version of stochastic gradient descent\nand optimized using the mean squared error, or ‘\nmse\n‘ loss function.\nOnce the model is defined, we can fit it on the training dataset.\n...\n# fit model\nmodel.fit(X, y, epochs=200, verbose=0)\n1\n2\n3\n.\n.\n.\n# fit model\nmodel\n.\nfit\n(\nX\n,\ny\n,\nepochs\n=\n200\n,\nverbose\n=\n0\n)\nAfter the model is fit, we can use it to make a prediction.\nWe can predict the next value in the sequence by providing the input:\n[70, 80, 90]\n1\n[70, 80, 90]\nAnd expecting the model to predict something like:\n[100]\n1\n[100]\nThe model expects the input shape to be three-dimensional with [\nsamples, timesteps, features\n], therefore, we must reshape the single input sample before making the prediction.\n...\n# demonstrate prediction\nx_input = array([70, 80, 90])\nx_input = x_input.reshape((1, n_steps, n_features))\nyhat = model.predict(x_input, verbose=0)\n1\n2\n3\n4\n5\n.\n.\n.\n# demonstrate prediction\nx_input\n=\narray\n(\n[\n70\n,\n80\n,\n90\n]\n)\nx_input\n=\nx_input\n.\nreshape\n(\n(\n1\n,\nn_steps\n,\nn_features\n)\n)\nyhat\n=\nmodel\n.\npredict\n(\nx_input\n,\nverbose\n=\n0\n)\nWe can tie all of this together and demonstrate how to develop a Vanilla LSTM for univariate time series forecasting and make a single prediction.\n# univariate lstm example\nfrom numpy import array\nfrom keras.models import Sequential\nfrom keras.layers import LSTM\nfrom keras.layers import Dense\n\n# split a univariate sequence into samples\ndef split_sequence(sequence, n_steps):\n\tX, y = list(), list()\n\tfor i in range(len(sequence)):\n\t\t# find the end of this pattern\n\t\tend_ix = i + n_steps\n\t\t# check if we are beyond the sequence\n\t\tif end_ix > len(sequence)-1:\n\t\t\tbreak\n\t\t# gather input and output parts of the pattern\n\t\tseq_x, seq_y = sequence[i:end_ix], sequence[end_ix]\n\t\tX.append(seq_x)\n\t\ty.append(seq_y)\n\treturn array(X), array(y)\n\n# define input sequence\nraw_seq = [10, 20, 30, 40, 50, 60, 70, 80, 90]\n# choose a number of time steps\nn_steps = 3\n# split into samples\nX, y = split_sequence(raw_seq, n_steps)\n# reshape from [samples, timesteps] into [samples, timesteps, features]\nn_features = 1\nX = X.reshape((X.shape[0], X.shape[1], n_features))\n# define model\nmodel = Sequential()\nmodel.add(LSTM(50, activation='relu', input_shape=(n_steps, n_features)))\nmodel.add(Dense(1))\nmodel.compile(optimizer='adam', loss='mse')\n# fit model\nmodel.fit(X, y, epochs=200, verbose=0)\n# demonstrate prediction\nx_input = array([70, 80, 90])\nx_input = x_input.reshape((1, n_steps, n_features))\nyhat = model.predict(x_input, verbose=0)\nprint(yhat)\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n14\n15\n16\n17\n18\n19\n20\n21\n22\n23\n24\n25\n26\n27\n28\n29\n30\n31\n32\n33\n34\n35\n36\n37\n38\n39\n40\n41\n42\n# univariate lstm example\nfrom\nnumpy\nimport\narray\nfrom\nkeras\n.\nmodels\nimport\nSequential\nfrom\nkeras\n.\nlayers\nimport\nLSTM\nfrom\nkeras\n.\nlayers\nimport\nDense\n# split a univariate sequence into samples\ndef\nsplit_sequence\n(\nsequence\n,\nn_steps\n)\n:\nX\n,\ny\n=\nlist\n(\n)\n,\nlist\n(\n)\nfor\ni\nin\nrange\n(\nlen\n(\nsequence\n)\n)\n:\n# find the end of this pattern\nend_ix\n=\ni\n+\nn\n_\nsteps\n# check if we are beyond the sequence\nif\nend_ix\n>\nlen\n(\nsequence\n)\n-\n1\n:\nbreak\n# gather input and output parts of the pattern\nseq_x\n,\nseq_y\n=\nsequence\n[\ni\n:\nend_ix\n]\n,\nsequence\n[\nend_ix\n]\nX\n.\nappend\n(\nseq_x\n)\ny\n.\nappend\n(\nseq_y\n)\nreturn\narray\n(\nX\n)\n,\narray\n(\ny\n)\n# define input sequence\nraw_seq\n=\n[\n10\n,\n20\n,\n30\n,\n40\n,\n50\n,\n60\n,\n70\n,\n80\n,\n90\n]\n# choose a number of time steps\nn_steps\n=\n3\n# split into samples\nX\n,\ny\n=\nsplit_sequence\n(\nraw_seq\n,\nn_steps\n)\n# reshape from [samples, timesteps] into [samples, timesteps, features]\nn_features\n=\n1\nX\n=\nX\n.\nreshape\n(\n(\nX\n.\nshape\n[\n0\n]\n,\nX\n.\nshape\n[\n1\n]\n,\nn_features\n)\n)\n# define model\nmodel\n=\nSequential\n(\n)\nmodel\n.\nadd\n(\nLSTM\n(\n50\n,\nactivation\n=\n'relu'\n,\ninput_shape\n=\n(\nn_steps\n,\nn_features\n)\n)\n)\nmodel\n.\nadd\n(\nDense\n(\n1\n)\n)\nmodel\n.\ncompile\n(\noptimizer\n=\n'adam'\n,\nloss\n=\n'mse'\n)\n# fit model\nmodel\n.\nfit\n(\nX\n,\ny\n,\nepochs\n=\n200\n,\nverbose\n=\n0\n)\n# demonstrate prediction\nx_input\n=\narray\n(\n[\n70\n,\n80\n,\n90\n]\n)\nx_input\n=\nx_input\n.\nreshape\n(\n(\n1\n,\nn_steps\n,\nn_features\n)\n)\nyhat\n=\nmodel\n.\npredict\n(\nx_input\n,\nverbose\n=\n0\n)\nprint\n(\nyhat\n)\nRunning the example prepares the data, fits the model, and makes a prediction.\nNote\n: Your\nresults may vary\ngiven the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.\nWe can see that the model predicts the next value in the sequence.\n[[102.09213]]\n1\n[[102.09213]]\nStacked LSTM\nMultiple hidden LSTM layers can be stacked one on top of another in what is referred to as a Stacked LSTM model.\nAn LSTM layer requires a three-dimensional input and LSTMs by default will produce a two-dimensional output as an interpretation from the end of the sequence.\nWe can address this by having the LSTM output a value for each time step in the input data by setting the\nreturn_sequences=True\nargument on the layer. This allows us to have 3D output from hidden LSTM layer as input to the next.\nWe can therefore define a Stacked LSTM as follows.\n...\n# define model\nmodel = Sequential()\nmodel.add(LSTM(50, activation='relu', return_sequences=True, input_shape=(n_steps, n_features)))\nmodel.add(LSTM(50, activation='relu'))\nmodel.add(Dense(1))\nmodel.compile(optimizer='adam', loss='mse')\n1\n2\n3\n4\n5\n6\n7\n.\n.\n.\n# define model\nmodel\n=\nSequential\n(\n)\nmodel\n.\nadd\n(\nLSTM\n(\n50\n,\nactivation\n=\n'relu'\n,\nreturn_sequences\n=\nTrue\n,\ninput_shape\n=\n(\nn_steps\n,\nn_features\n)\n)\n)\nmodel\n.\nadd\n(\nLSTM\n(\n50\n,\nactivation\n=\n'relu'\n)\n)\nmodel\n.\nadd\n(\nDense\n(\n1\n)\n)\nmodel\n.\ncompile\n(\noptimizer\n=\n'adam'\n,\nloss\n=\n'mse'\n)\nWe can tie this together; the complete code example is listed below.\n# univariate stacked lstm example\nfrom numpy import array\nfrom keras.models import Sequential\nfrom keras.layers import LSTM\nfrom keras.layers import Dense\n\n# split a univariate sequence\ndef split_sequence(sequence, n_steps):\n\tX, y = list(), list()\n\tfor i in range(len(sequence)):\n\t\t# find the end of this pattern\n\t\tend_ix = i + n_steps\n\t\t# check if we are beyond the sequence\n\t\tif end_ix > len(sequence)-1:\n\t\t\tbreak\n\t\t# gather input and output parts of the pattern\n\t\tseq_x, seq_y = sequence[i:end_ix], sequence[end_ix]\n\t\tX.append(seq_x)\n\t\ty.append(seq_y)\n\treturn array(X), array(y)\n\n# define input sequence\nraw_seq = [10, 20, 30, 40, 50, 60, 70, 80, 90]\n# choose a number of time steps\nn_steps = 3\n# split into samples\nX, y = split_sequence(raw_seq, n_steps)\n# reshape from [samples, timesteps] into [samples, timesteps, features]\nn_features = 1\nX = X.reshape((X.shape[0], X.shape[1], n_features))\n# define model\nmodel = Sequential()\nmodel.add(LSTM(50, activation='relu', return_sequences=True, input_shape=(n_steps, n_features)))\nmodel.add(LSTM(50, activation='relu'))\nmodel.add(Dense(1))\nmodel.compile(optimizer='adam', loss='mse')\n# fit model\nmodel.fit(X, y, epochs=200, verbose=0)\n# demonstrate prediction\nx_input = array([70, 80, 90])\nx_input = x_input.reshape((1, n_steps, n_features))\nyhat = model.predict(x_input, verbose=0)\nprint(yhat)\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n14\n15\n16\n17\n18\n19\n20\n21\n22\n23\n24\n25\n26\n27\n28\n29\n30\n31\n32\n33\n34\n35\n36\n37\n38\n39\n40\n41\n42\n43\n# univariate stacked lstm example\nfrom\nnumpy\nimport\narray\nfrom\nkeras\n.\nmodels\nimport\nSequential\nfrom\nkeras\n.\nlayers\nimport\nLSTM\nfrom\nkeras\n.\nlayers\nimport\nDense\n# split a univariate sequence\ndef\nsplit_sequence\n(\nsequence\n,\nn_steps\n)\n:\nX\n,\ny\n=\nlist\n(\n)\n,\nlist\n(\n)\nfor\ni\nin\nrange\n(\nlen\n(\nsequence\n)\n)\n:\n# find the end of this pattern\nend_ix\n=\ni\n+\nn\n_\nsteps\n# check if we are beyond the sequence\nif\nend_ix\n>\nlen\n(\nsequence\n)\n-\n1\n:\nbreak\n# gather input and output parts of the pattern\nseq_x\n,\nseq_y\n=\nsequence\n[\ni\n:\nend_ix\n]\n,\nsequence\n[\nend_ix\n]\nX\n.\nappend\n(\nseq_x\n)\ny\n.\nappend\n(\nseq_y\n)\nreturn\narray\n(\nX\n)\n,\narray\n(\ny\n)\n# define input sequence\nraw_seq\n=\n[\n10\n,\n20\n,\n30\n,\n40\n,\n50\n,\n60\n,\n70\n,\n80\n,\n90\n]\n# choose a number of time steps\nn_steps\n=\n3\n# split into samples\nX\n,\ny\n=\nsplit_sequence\n(\nraw_seq\n,\nn_steps\n)\n# reshape from [samples, timesteps] into [samples, timesteps, features]\nn_features\n=\n1\nX\n=\nX\n.\nreshape\n(\n(\nX\n.\nshape\n[\n0\n]\n,\nX\n.\nshape\n[\n1\n]\n,\nn_features\n)\n)\n# define model\nmodel\n=\nSequential\n(\n)\nmodel\n.\nadd\n(\nLSTM\n(\n50\n,\nactivation\n=\n'relu'\n,\nreturn_sequences\n=\nTrue\n,\ninput_shape\n=\n(\nn_steps\n,\nn_features\n)\n)\n)\nmodel\n.\nadd\n(\nLSTM\n(\n50\n,\nactivation\n=\n'relu'\n)\n)\nmodel\n.\nadd\n(\nDense\n(\n1\n)\n)\nmodel\n.\ncompile\n(\noptimizer\n=\n'adam'\n,\nloss\n=\n'mse'\n)\n# fit model\nmodel\n.\nfit\n(\nX\n,\ny\n,\nepochs\n=\n200\n,\nverbose\n=\n0\n)\n# demonstrate prediction\nx_input\n=\narray\n(\n[\n70\n,\n80\n,\n90\n]\n)\nx_input\n=\nx_input\n.\nreshape\n(\n(\n1\n,\nn_steps\n,\nn_features\n)\n)\nyhat\n=\nmodel\n.\npredict\n(\nx_input\n,\nverbose\n=\n0\n)\nprint\n(\nyhat\n)\nNote\n: Your\nresults may vary\ngiven the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.\nRunning the example predicts the next value in the sequence, which we expect would be 100.\n[[102.47341]]\n1\n[[102.47341]]\nBidirectional LSTM\nOn some sequence prediction problems, it can be beneficial to allow the LSTM model to learn the input sequence both forward and backwards and concatenate both interpretations.\nThis is called a\nBidirectional LSTM\n.\nWe can implement a Bidirectional LSTM for univariate time series forecasting by wrapping the first hidden layer in a wrapper layer called Bidirectional.\nAn example of defining a Bidirectional LSTM to read input both forward and backward is as follows.\n...\n# define model\nmodel = Sequential()\nmodel.add(Bidirectional(LSTM(50, activation='relu'), input_shape=(n_steps, n_features)))\nmodel.add(Dense(1))\nmodel.compile(optimizer='adam', loss='mse')\n1\n2\n3\n4\n5\n6\n.\n.\n.\n# define model\nmodel\n=\nSequential\n(\n)\nmodel\n.\nadd\n(\nBidirectional\n(\nLSTM\n(\n50\n,\nactivation\n=\n'relu'\n)\n,\ninput_shape\n=\n(\nn_steps\n,\nn_features\n)\n)\n)\nmodel\n.\nadd\n(\nDense\n(\n1\n)\n)\nmodel\n.\ncompile\n(\noptimizer\n=\n'adam'\n,\nloss\n=\n'mse'\n)\nThe complete example of the Bidirectional LSTM for univariate time series forecasting is listed below.\n# univariate bidirectional lstm example\nfrom numpy import array\nfrom keras.models import Sequential\nfrom keras.layers import LSTM\nfrom keras.layers import Dense\nfrom keras.layers import Bidirectional\n\n# split a univariate sequence\ndef split_sequence(sequence, n_steps):\n\tX, y = list(), list()\n\tfor i in range(len(sequence)):\n\t\t# find the end of this pattern\n\t\tend_ix = i + n_steps\n\t\t# check if we are beyond the sequence\n\t\tif end_ix > len(sequence)-1:\n\t\t\tbreak\n\t\t# gather input and output parts of the pattern\n\t\tseq_x, seq_y = sequence[i:end_ix], sequence[end_ix]\n\t\tX.append(seq_x)\n\t\ty.append(seq_y)\n\treturn array(X), array(y)\n\n# define input sequence\nraw_seq = [10, 20, 30, 40, 50, 60, 70, 80, 90]\n# choose a number of time steps\nn_steps = 3\n# split into samples\nX, y = split_sequence(raw_seq, n_steps)\n# reshape from [samples, timesteps] into [samples, timesteps, features]\nn_features = 1\nX = X.reshape((X.shape[0], X.shape[1], n_features))\n# define model\nmodel = Sequential()\nmodel.add(Bidirectional(LSTM(50, activation='relu'), input_shape=(n_steps, n_features)))\nmodel.add(Dense(1))\nmodel.compile(optimizer='adam', loss='mse')\n# fit model\nmodel.fit(X, y, epochs=200, verbose=0)\n# demonstrate prediction\nx_input = array([70, 80, 90])\nx_input = x_input.reshape((1, n_steps, n_features))\nyhat = model.predict(x_input, verbose=0)\nprint(yhat)\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n14\n15\n16\n17\n18\n19\n20\n21\n22\n23\n24\n25\n26\n27\n28\n29\n30\n31\n32\n33\n34\n35\n36\n37\n38\n39\n40\n41\n42\n43\n# univariate bidirectional lstm example\nfrom\nnumpy\nimport\narray\nfrom\nkeras\n.\nmodels\nimport\nSequential\nfrom\nkeras\n.\nlayers\nimport\nLSTM\nfrom\nkeras\n.\nlayers\nimport\nDense\nfrom\nkeras\n.\nlayers\nimport\nBidirectional\n# split a univariate sequence\ndef\nsplit_sequence\n(\nsequence\n,\nn_steps\n)\n:\nX\n,\ny\n=\nlist\n(\n)\n,\nlist\n(\n)\nfor\ni\nin\nrange\n(\nlen\n(\nsequence\n)\n)\n:\n# find the end of this pattern\nend_ix\n=\ni\n+\nn\n_\nsteps\n# check if we are beyond the sequence\nif\nend_ix\n>\nlen\n(\nsequence\n)\n-\n1\n:\nbreak\n# gather input and output parts of the pattern\nseq_x\n,\nseq_y\n=\nsequence\n[\ni\n:\nend_ix\n]\n,\nsequence\n[\nend_ix\n]\nX\n.\nappend\n(\nseq_x\n)\ny\n.\nappend\n(\nseq_y\n)\nreturn\narray\n(\nX\n)\n,\narray\n(\ny\n)\n# define input sequence\nraw_seq\n=\n[\n10\n,\n20\n,\n30\n,\n40\n,\n50\n,\n60\n,\n70\n,\n80\n,\n90\n]\n# choose a number of time steps\nn_steps\n=\n3\n# split into samples\nX\n,\ny\n=\nsplit_sequence\n(\nraw_seq\n,\nn_steps\n)\n# reshape from [samples, timesteps] into [samples, timesteps, features]\nn_features\n=\n1\nX\n=\nX\n.\nreshape\n(\n(\nX\n.\nshape\n[\n0\n]\n,\nX\n.\nshape\n[\n1\n]\n,\nn_features\n)\n)\n# define model\nmodel\n=\nSequential\n(\n)\nmodel\n.\nadd\n(\nBidirectional\n(\nLSTM\n(\n50\n,\nactivation\n=\n'relu'\n)\n,\ninput_shape\n=\n(\nn_steps\n,\nn_features\n)\n)\n)\nmodel\n.\nadd\n(\nDense\n(\n1\n)\n)\nmodel\n.\ncompile\n(\noptimizer\n=\n'adam'\n,\nloss\n=\n'mse'\n)\n# fit model\nmodel\n.\nfit\n(\nX\n,\ny\n,\nepochs\n=\n200\n,\nverbose\n=\n0\n)\n# demonstrate prediction\nx_input\n=\narray\n(\n[\n70\n,\n80\n,\n90\n]\n)\nx_input\n=\nx_input\n.\nreshape\n(\n(\n1\n,\nn_steps\n,\nn_features\n)\n)\nyhat\n=\nmodel\n.\npredict\n(\nx_input\n,\nverbose\n=\n0\n)\nprint\n(\nyhat\n)\nNote\n: Your\nresults may vary\ngiven the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.\nRunning the example predicts the next value in the sequence, which we expect would be 100.\n[[101.48093]]\n1\n[[101.48093]]\nCNN LSTM\nA convolutional neural network, or CNN for short, is a type of neural network developed for working with two-dimensional image data.\nThe CNN can be very effective at automatically extracting and learning features from one-dimensional sequence data such as univariate time series data.\nA CNN model can be used in a hybrid model with an LSTM backend where the CNN is used to interpret subsequences of input that together are provided as a sequence to an LSTM model to interpret.\nThis hybrid model is called a CNN-LSTM\n.\nThe first step is to split the input sequences into subsequences that can be processed by the CNN model. For example, we can first split our univariate time series data into input/output samples with four steps as input and one as output. Each sample can then be split into two sub-samples, each with two time steps. The CNN can interpret each subsequence of two time steps and provide a time series of interpretations of the subsequences to the LSTM model to process as input.\nWe can parameterize this and define the number of subsequences as\nn_seq\nand the number of time steps per subsequence as\nn_steps\n. The input data can then be reshaped to have the required structure:\n[samples, subsequences, timesteps, features]\n1\n[samples, subsequences, timesteps, features]\nFor example:\n...\n# choose a number of time steps\nn_steps = 4\n# split into samples\nX, y = split_sequence(raw_seq, n_steps)\n# reshape from [samples, timesteps] into [samples, subsequences, timesteps, features]\nn_features = 1\nn_seq = 2\nn_steps = 2\nX = X.reshape((X.shape[0], n_seq, n_steps, n_features))\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n.\n.\n.\n# choose a number of time steps\nn_steps\n=\n4\n# split into samples\nX\n,\ny\n=\nsplit_sequence\n(\nraw_seq\n,\nn_steps\n)\n# reshape from [samples, timesteps] into [samples, subsequences, timesteps, features]\nn_features\n=\n1\nn_seq\n=\n2\nn_steps\n=\n2\nX\n=\nX\n.\nreshape\n(\n(\nX\n.\nshape\n[\n0\n]\n,\nn_seq\n,\nn_steps\n,\nn_features\n)\n)\nWe want to reuse the same CNN model when reading in each sub-sequence of data separately.\nThis can be achieved by wrapping the entire CNN model in a\nTimeDistributed wrapper\nthat will apply the entire model once per input, in this case, once per input subsequence.\nThe CNN model first has a convolutional layer for reading across the subsequence that requires a number of filters and a kernel size to be specified. The number of filters is the number of reads or interpretations of the input sequence. The kernel size is the number of time steps included of each ‘read’ operation of the input sequence.\nThe convolution layer is followed by a max pooling layer that distills the filter maps down to 1/2 of their size that includes the most salient features. These structures are then flattened down to a single one-dimensional vector to be used as a single input time step to the LSTM layer.\n...\nmodel.add(TimeDistributed(Conv1D(filters=64, kernel_size=1, activation='relu'), input_shape=(None, n_steps, n_features)))\nmodel.add(TimeDistributed(MaxPooling1D(pool_size=2)))\nmodel.add(TimeDistributed(Flatten()))\n1\n2\n3\n4\n.\n.\n.\nmodel\n.\nadd\n(\nTimeDistributed\n(\nConv1D\n(\nfilters\n=\n64\n,\nkernel_size\n=\n1\n,\nactivation\n=\n'relu'\n)\n,\ninput_shape\n=\n(\nNone\n,\nn_steps\n,\nn_features\n)\n)\n)\nmodel\n.\nadd\n(\nTimeDistributed\n(\nMaxPooling1D\n(\npool_size\n=\n2\n)\n)\n)\nmodel\n.\nadd\n(\nTimeDistributed\n(\nFlatten\n(\n)\n)\n)\nNext, we can define the LSTM part of the model that interprets the CNN model’s read of the input sequence and makes a prediction.\n...\nmodel.add(LSTM(50, activation='relu'))\nmodel.add(Dense(1))\n1\n2\n3\n.\n.\n.\nmodel\n.\nadd\n(\nLSTM\n(\n50\n,\nactivation\n=\n'relu'\n)\n)\nmodel\n.\nadd\n(\nDense\n(\n1\n)\n)\nWe can tie all of this together; the complete example of a CNN-LSTM model for univariate time series forecasting is listed below.\n# univariate cnn lstm example\nfrom numpy import array\nfrom keras.models import Sequential\nfrom keras.layers import LSTM\nfrom keras.layers import Dense\nfrom keras.layers import Flatten\nfrom keras.layers import TimeDistributed\nfrom keras.layers.convolutional import Conv1D\nfrom keras.layers.convolutional import MaxPooling1D\n\n# split a univariate sequence into samples\ndef split_sequence(sequence, n_steps):\n\tX, y = list(), list()\n\tfor i in range(len(sequence)):\n\t\t# find the end of this pattern\n\t\tend_ix = i + n_steps\n\t\t# check if we are beyond the sequence\n\t\tif end_ix > len(sequence)-1:\n\t\t\tbreak\n\t\t# gather input and output parts of the pattern\n\t\tseq_x, seq_y = sequence[i:end_ix], sequence[end_ix]\n\t\tX.append(seq_x)\n\t\ty.append(seq_y)\n\treturn array(X), array(y)\n\n# define input sequence\nraw_seq = [10, 20, 30, 40, 50, 60, 70, 80, 90]\n# choose a number of time steps\nn_steps = 4\n# split into samples\nX, y = split_sequence(raw_seq, n_steps)\n# reshape from [samples, timesteps] into [samples, subsequences, timesteps, features]\nn_features = 1\nn_seq = 2\nn_steps = 2\nX = X.reshape((X.shape[0], n_seq, n_steps, n_features))\n# define model\nmodel = Sequential()\nmodel.add(TimeDistributed(Conv1D(filters=64, kernel_size=1, activation='relu'), input_shape=(None, n_steps, n_features)))\nmodel.add(TimeDistributed(MaxPooling1D(pool_size=2)))\nmodel.add(TimeDistributed(Flatten()))\nmodel.add(LSTM(50, activation='relu'))\nmodel.add(Dense(1))\nmodel.compile(optimizer='adam', loss='mse')\n# fit model\nmodel.fit(X, y, epochs=500, verbose=0)\n# demonstrate prediction\nx_input = array([60, 70, 80, 90])\nx_input = x_input.reshape((1, n_seq, n_steps, n_features))\nyhat = model.predict(x_input, verbose=0)\nprint(yhat)\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n14\n15\n16\n17\n18\n19\n20\n21\n22\n23\n24\n25\n26\n27\n28\n29\n30\n31\n32\n33\n34\n35\n36\n37\n38\n39\n40\n41\n42\n43\n44\n45\n46\n47\n48\n49\n50\n51\n# univariate cnn lstm example\nfrom\nnumpy\nimport\narray\nfrom\nkeras\n.\nmodels\nimport\nSequential\nfrom\nkeras\n.\nlayers\nimport\nLSTM\nfrom\nkeras\n.\nlayers\nimport\nDense\nfrom\nkeras\n.\nlayers\nimport\nFlatten\nfrom\nkeras\n.\nlayers\nimport\nTimeDistributed\nfrom\nkeras\n.\nlayers\n.\nconvolutional\nimport\nConv1D\nfrom\nkeras\n.\nlayers\n.\nconvolutional\nimport\nMaxPooling1D\n# split a univariate sequence into samples\ndef\nsplit_sequence\n(\nsequence\n,\nn_steps\n)\n:\nX\n,\ny\n=\nlist\n(\n)\n,\nlist\n(\n)\nfor\ni\nin\nrange\n(\nlen\n(\nsequence\n)\n)\n:\n# find the end of this pattern\nend_ix\n=\ni\n+\nn\n_\nsteps\n# check if we are beyond the sequence\nif\nend_ix\n>\nlen\n(\nsequence\n)\n-\n1\n:\nbreak\n# gather input and output parts of the pattern\nseq_x\n,\nseq_y\n=\nsequence\n[\ni\n:\nend_ix\n]\n,\nsequence\n[\nend_ix\n]\nX\n.\nappend\n(\nseq_x\n)\ny\n.\nappend\n(\nseq_y\n)\nreturn\narray\n(\nX\n)\n,\narray\n(\ny\n)\n# define input sequence\nraw_seq\n=\n[\n10\n,\n20\n,\n30\n,\n40\n,\n50\n,\n60\n,\n70\n,\n80\n,\n90\n]\n# choose a number of time steps\nn_steps\n=\n4\n# split into samples\nX\n,\ny\n=\nsplit_sequence\n(\nraw_seq\n,\nn_steps\n)\n# reshape from [samples, timesteps] into [samples, subsequences, timesteps, features]\nn_features\n=\n1\nn_seq\n=\n2\nn_steps\n=\n2\nX\n=\nX\n.\nreshape\n(\n(\nX\n.\nshape\n[\n0\n]\n,\nn_seq\n,\nn_steps\n,\nn_features\n)\n)\n# define model\nmodel\n=\nSequential\n(\n)\nmodel\n.\nadd\n(\nTimeDistributed\n(\nConv1D\n(\nfilters\n=\n64\n,\nkernel_size\n=\n1\n,\nactivation\n=\n'relu'\n)\n,\ninput_shape\n=\n(\nNone\n,\nn_steps\n,\nn_features\n)\n)\n)\nmodel\n.\nadd\n(\nTimeDistributed\n(\nMaxPooling1D\n(\npool_size\n=\n2\n)\n)\n)\nmodel\n.\nadd\n(\nTimeDistributed\n(\nFlatten\n(\n)\n)\n)\nmodel\n.\nadd\n(\nLSTM\n(\n50\n,\nactivation\n=\n'relu'\n)\n)\nmodel\n.\nadd\n(\nDense\n(\n1\n)\n)\nmodel\n.\ncompile\n(\noptimizer\n=\n'adam'\n,\nloss\n=\n'mse'\n)\n# fit model\nmodel\n.\nfit\n(\nX\n,\ny\n,\nepochs\n=\n500\n,\nverbose\n=\n0\n)\n# demonstrate prediction\nx_input\n=\narray\n(\n[\n60\n,\n70\n,\n80\n,\n90\n]\n)\nx_input\n=\nx_input\n.\nreshape\n(\n(\n1\n,\nn_seq\n,\nn_steps\n,\nn_features\n)\n)\nyhat\n=\nmodel\n.\npredict\n(\nx_input\n,\nverbose\n=\n0\n)\nprint\n(\nyhat\n)\nNote\n: Your\nresults may vary\ngiven the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.\nRunning the example predicts the next value in the sequence, which we expect would be 100.\n[[101.69263]]\n1\n[[101.69263]]\nConvLSTM\nA type of LSTM related to the CNN-LSTM is the ConvLSTM, where the convolutional reading of input is built directly into each LSTM unit.\nThe ConvLSTM was developed for reading two-dimensional spatial-temporal data, but can be adapted for use with univariate time series forecasting.\nThe layer expects input as a sequence of two-dimensional images, therefore the shape of input data must be:\n[samples, timesteps, rows, columns, features]\n1\n[samples, timesteps, rows, columns, features]\nFor our purposes, we can split each sample into subsequences where timesteps will become the number of subsequences, or\nn_seq\n, and columns will be the number of time steps for each subsequence, or\nn_steps\n. The number of rows is fixed at 1 as we are working with one-dimensional data.\nWe can now reshape the prepared samples into the required structure.\n...\n# choose a number of time steps\nn_steps = 4\n# split into samples\nX, y = split_sequence(raw_seq, n_steps)\n# reshape from [samples, timesteps] into [samples, timesteps, rows, columns, features]\nn_features = 1\nn_seq = 2\nn_steps = 2\nX = X.reshape((X.shape[0], n_seq, 1, n_steps, n_features))\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n.\n.\n.\n# choose a number of time steps\nn_steps\n=\n4\n# split into samples\nX\n,\ny\n=\nsplit_sequence\n(\nraw_seq\n,\nn_steps\n)\n# reshape from [samples, timesteps] into [samples, timesteps, rows, columns, features]\nn_features\n=\n1\nn_seq\n=\n2\nn_steps\n=\n2\nX\n=\nX\n.\nreshape\n(\n(\nX\n.\nshape\n[\n0\n]\n,\nn_seq\n,\n1\n,\nn_steps\n,\nn_features\n)\n)\nWe can define the ConvLSTM as a single layer in terms of the number of filters and a two-dimensional kernel size in terms of (rows, columns). As we are working with a one-dimensional series, the number of rows is always fixed to 1 in the kernel.\nThe output of the model must then be flattened before it can be interpreted and a prediction made.\n...\nmodel.add(ConvLSTM2D(filters=64, kernel_size=(1,2), activation='relu', input_shape=(n_seq, 1, n_steps, n_features)))\nmodel.add(Flatten())\n1\n2\n3\n.\n.\n.\nmodel\n.\nadd\n(\nConvLSTM2D\n(\nfilters\n=\n64\n,\nkernel_size\n=\n(\n1\n,\n2\n)\n,\nactivation\n=\n'relu'\n,\ninput_shape\n=\n(\nn_seq\n,\n1\n,\nn_steps\n,\nn_features\n)\n)\n)\nmodel\n.\nadd\n(\nFlatten\n(\n)\n)\nThe complete example of a ConvLSTM for one-step univariate time series forecasting is listed below.\n# univariate convlstm example\nfrom numpy import array\nfrom keras.models import Sequential\nfrom keras.layers import LSTM\nfrom keras.layers import Dense\nfrom keras.layers import Flatten\nfrom keras.layers import ConvLSTM2D\n\n# split a univariate sequence into samples\ndef split_sequence(sequence, n_steps):\n\tX, y = list(), list()\n\tfor i in range(len(sequence)):\n\t\t# find the end of this pattern\n\t\tend_ix = i + n_steps\n\t\t# check if we are beyond the sequence\n\t\tif end_ix > len(sequence)-1:\n\t\t\tbreak\n\t\t# gather input and output parts of the pattern\n\t\tseq_x, seq_y = sequence[i:end_ix], sequence[end_ix]\n\t\tX.append(seq_x)\n\t\ty.append(seq_y)\n\treturn array(X), array(y)\n\n# define input sequence\nraw_seq = [10, 20, 30, 40, 50, 60, 70, 80, 90]\n# choose a number of time steps\nn_steps = 4\n# split into samples\nX, y = split_sequence(raw_seq, n_steps)\n# reshape from [samples, timesteps] into [samples, timesteps, rows, columns, features]\nn_features = 1\nn_seq = 2\nn_steps = 2\nX = X.reshape((X.shape[0], n_seq, 1, n_steps, n_features))\n# define model\nmodel = Sequential()\nmodel.add(ConvLSTM2D(filters=64, kernel_size=(1,2), activation='relu', input_shape=(n_seq, 1, n_steps, n_features)))\nmodel.add(Flatten())\nmodel.add(Dense(1))\nmodel.compile(optimizer='adam', loss='mse')\n# fit model\nmodel.fit(X, y, epochs=500, verbose=0)\n# demonstrate prediction\nx_input = array([60, 70, 80, 90])\nx_input = x_input.reshape((1, n_seq, 1, n_steps, n_features))\nyhat = model.predict(x_input, verbose=0)\nprint(yhat)\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n14\n15\n16\n17\n18\n19\n20\n21\n22\n23\n24\n25\n26\n27\n28\n29\n30\n31\n32\n33\n34\n35\n36\n37\n38\n39\n40\n41\n42\n43\n44\n45\n46\n47\n# univariate convlstm example\nfrom\nnumpy\nimport\narray\nfrom\nkeras\n.\nmodels\nimport\nSequential\nfrom\nkeras\n.\nlayers\nimport\nLSTM\nfrom\nkeras\n.\nlayers\nimport\nDense\nfrom\nkeras\n.\nlayers\nimport\nFlatten\nfrom\nkeras\n.\nlayers\nimport\nConvLSTM2D\n# split a univariate sequence into samples\ndef\nsplit_sequence\n(\nsequence\n,\nn_steps\n)\n:\nX\n,\ny\n=\nlist\n(\n)\n,\nlist\n(\n)\nfor\ni\nin\nrange\n(\nlen\n(\nsequence\n)\n)\n:\n# find the end of this pattern\nend_ix\n=\ni\n+\nn\n_\nsteps\n# check if we are beyond the sequence\nif\nend_ix\n>\nlen\n(\nsequence\n)\n-\n1\n:\nbreak\n# gather input and output parts of the pattern\nseq_x\n,\nseq_y\n=\nsequence\n[\ni\n:\nend_ix\n]\n,\nsequence\n[\nend_ix\n]\nX\n.\nappend\n(\nseq_x\n)\ny\n.\nappend\n(\nseq_y\n)\nreturn\narray\n(\nX\n)\n,\narray\n(\ny\n)\n# define input sequence\nraw_seq\n=\n[\n10\n,\n20\n,\n30\n,\n40\n,\n50\n,\n60\n,\n70\n,\n80\n,\n90\n]\n# choose a number of time steps\nn_steps\n=\n4\n# split into samples\nX\n,\ny\n=\nsplit_sequence\n(\nraw_seq\n,\nn_steps\n)\n# reshape from [samples, timesteps] into [samples, timesteps, rows, columns, features]\nn_features\n=\n1\nn_seq\n=\n2\nn_steps\n=\n2\nX\n=\nX\n.\nreshape\n(\n(\nX\n.\nshape\n[\n0\n]\n,\nn_seq\n,\n1\n,\nn_steps\n,\nn_features\n)\n)\n# define model\nmodel\n=\nSequential\n(\n)\nmodel\n.\nadd\n(\nConvLSTM2D\n(\nfilters\n=\n64\n,\nkernel_size\n=\n(\n1\n,\n2\n)\n,\nactivation\n=\n'relu'\n,\ninput_shape\n=\n(\nn_seq\n,\n1\n,\nn_steps\n,\nn_features\n)\n)\n)\nmodel\n.\nadd\n(\nFlatten\n(\n)\n)\nmodel\n.\nadd\n(\nDense\n(\n1\n)\n)\nmodel\n.\ncompile\n(\noptimizer\n=\n'adam'\n,\nloss\n=\n'mse'\n)\n# fit model\nmodel\n.\nfit\n(\nX\n,\ny\n,\nepochs\n=\n500\n,\nverbose\n=\n0\n)\n# demonstrate prediction\nx_input\n=\narray\n(\n[\n60\n,\n70\n,\n80\n,\n90\n]\n)\nx_input\n=\nx_input\n.\nreshape\n(\n(\n1\n,\nn_seq\n,\n1\n,\nn_steps\n,\nn_features\n)\n)\nyhat\n=\nmodel\n.\npredict\n(\nx_input\n,\nverbose\n=\n0\n)\nprint\n(\nyhat\n)\nNote\n: Your\nresults may vary\ngiven the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.\nRunning the example predicts the next value in the sequence, which we expect would be 100.\n[[103.68166]]\n1\n[[103.68166]]\nNow that we have looked at LSTM models for univariate data, let’s turn our attention to multivariate data.\nMultivariate LSTM Models\nMultivariate time series data means data where there is more than one observation for each time step.\nThere are two main models that we may require with multivariate time series data; they are:\nMultiple Input Series.\nMultiple Parallel Series.\nLet’s take a look at each in turn.\nMultiple Input Series\nA problem may have two or more parallel input time series and an output time series that is dependent on the input time series.\nThe input time series are parallel because each series has an observation at the same time steps.\nWe can demonstrate this with a simple example of two parallel input time series where the output series is the simple addition of the input series.\n...\n# define input sequence\nin_seq1 = array([10, 20, 30, 40, 50, 60, 70, 80, 90])\nin_seq2 = array([15, 25, 35, 45, 55, 65, 75, 85, 95])\nout_seq = array([in_seq1[i]+in_seq2[i] for i in range(len(in_seq1))])\n1\n2\n3\n4\n5\n.\n.\n.\n# define input sequence\nin_seq1\n=\narray\n(\n[\n10\n,\n20\n,\n30\n,\n40\n,\n50\n,\n60\n,\n70\n,\n80\n,\n90\n]\n)\nin_seq2\n=\narray\n(\n[\n15\n,\n25\n,\n35\n,\n45\n,\n55\n,\n65\n,\n75\n,\n85\n,\n95\n]\n)\nout_seq\n=\narray\n(\n[\nin_seq1\n[\ni\n]\n+\nin_seq2\n[\ni\n]\nfor\ni\nin\nrange\n(\nlen\n(\nin_seq1\n)\n)\n]\n)\nWe can reshape these three arrays of data as a single dataset where each row is a time step, and each column is a separate time series. This is a standard way of storing parallel time series in a CSV file.\n...\n# convert to [rows, columns] structure\nin_seq1 = in_seq1.reshape((len(in_seq1), 1))\nin_seq2 = in_seq2.reshape((len(in_seq2), 1))\nout_seq = out_seq.reshape((len(out_seq), 1))\n# horizontally stack columns\ndataset = hstack((in_seq1, in_seq2, out_seq))\n1\n2\n3\n4\n5\n6\n7\n.\n.\n.\n# convert to [rows, columns] structure\nin_seq1\n=\nin_seq1\n.\nreshape\n(\n(\nlen\n(\nin_seq1\n)\n,\n1\n)\n)\nin_seq2\n=\nin_seq2\n.\nreshape\n(\n(\nlen\n(\nin_seq2\n)\n,\n1\n)\n)\nout_seq\n=\nout_seq\n.\nreshape\n(\n(\nlen\n(\nout_seq\n)\n,\n1\n)\n)\n# horizontally stack columns\ndataset\n=\nhstack\n(\n(\nin_seq1\n,\nin_seq2\n,\nout_seq\n)\n)\nThe complete example is listed below.\n# multivariate data preparation\nfrom numpy import array\nfrom numpy import hstack\n# define input sequence\nin_seq1 = array([10, 20, 30, 40, 50, 60, 70, 80, 90])\nin_seq2 = array([15, 25, 35, 45, 55, 65, 75, 85, 95])\nout_seq = array([in_seq1[i]+in_seq2[i] for i in range(len(in_seq1))])\n# convert to [rows, columns] structure\nin_seq1 = in_seq1.reshape((len(in_seq1), 1))\nin_seq2 = in_seq2.reshape((len(in_seq2), 1))\nout_seq = out_seq.reshape((len(out_seq), 1))\n# horizontally stack columns\ndataset = hstack((in_seq1, in_seq2, out_seq))\nprint(dataset)\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n14\n# multivariate data preparation\nfrom\nnumpy\nimport\narray\nfrom\nnumpy\nimport\nhstack\n# define input sequence\nin_seq1\n=\narray\n(\n[\n10\n,\n20\n,\n30\n,\n40\n,\n50\n,\n60\n,\n70\n,\n80\n,\n90\n]\n)\nin_seq2\n=\narray\n(\n[\n15\n,\n25\n,\n35\n,\n45\n,\n55\n,\n65\n,\n75\n,\n85\n,\n95\n]\n)\nout_seq\n=\narray\n(\n[\nin_seq1\n[\ni\n]\n+\nin_seq2\n[\ni\n]\nfor\ni\nin\nrange\n(\nlen\n(\nin_seq1\n)\n)\n]\n)\n# convert to [rows, columns] structure\nin_seq1\n=\nin_seq1\n.\nreshape\n(\n(\nlen\n(\nin_seq1\n)\n,\n1\n)\n)\nin_seq2\n=\nin_seq2\n.\nreshape\n(\n(\nlen\n(\nin_seq2\n)\n,\n1\n)\n)\nout_seq\n=\nout_seq\n.\nreshape\n(\n(\nlen\n(\nout_seq\n)\n,\n1\n)\n)\n# horizontally stack columns\ndataset\n=\nhstack\n(\n(\nin_seq1\n,\nin_seq2\n,\nout_seq\n)\n)\nprint\n(\ndataset\n)\nRunning the example prints the dataset with one row per time step and one column for each of the two input and one output parallel time series.\n[[ 10  15  25]\n [ 20  25  45]\n [ 30  35  65]\n [ 40  45  85]\n [ 50  55 105]\n [ 60  65 125]\n [ 70  75 145]\n [ 80  85 165]\n [ 90  95 185]]\n1\n2\n3\n4\n5\n6\n7\n8\n9\n[[ 10  15  25]\n[ 20  25  45]\n[ 30  35  65]\n[ 40  45  85]\n[ 50  55 105]\n[ 60  65 125]\n[ 70  75 145]\n[ 80  85 165]\n[ 90  95 185]]\nAs with the univariate time series, we must structure these data into samples with input and output elements.\nAn LSTM model needs sufficient context to learn a mapping from an input sequence to an output value. LSTMs can support parallel input time series as separate variables or features. Therefore, we need to split the data into samples maintaining the order of observations across the two input sequences.\nIf we chose three input time steps, then the first sample would look as follows:\nInput:\n10, 15\n20, 25\n30, 35\n1\n2\n3\n10, 15\n20, 25\n30, 35\nOutput:\n65\n1\n65\nThat is, the first three time steps of each parallel series are provided as input to the model and the model associates this with the value in the output series at the third time step, in this case, 65.\nWe can see that, in transforming the time series into input/output samples to train the model, that we will have to discard some values from the output time series where we do not have values in the input time series at prior time steps. In turn, the choice of the size of the number of input time steps will have an important effect on how much of the training data is used.\nWe can define a function named\nsplit_sequences()\nthat will take a dataset as we have defined it with rows for time steps and columns for parallel series and return input/output samples.\n# split a multivariate sequence into samples\ndef split_sequences(sequences, n_steps):\n\tX, y = list(), list()\n\tfor i in range(len(sequences)):\n\t\t# find the end of this pattern\n\t\tend_ix = i + n_steps\n\t\t# check if we are beyond the dataset\n\t\tif end_ix > len(sequences):\n\t\t\tbreak\n\t\t# gather input and output parts of the pattern\n\t\tseq_x, seq_y = sequences[i:end_ix, :-1], sequences[end_ix-1, -1]\n\t\tX.append(seq_x)\n\t\ty.append(seq_y)\n\treturn array(X), array(y)\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n14\n# split a multivariate sequence into samples\ndef\nsplit_sequences\n(\nsequences\n,\nn_steps\n)\n:\nX\n,\ny\n=\nlist\n(\n)\n,\nlist\n(\n)\nfor\ni\nin\nrange\n(\nlen\n(\nsequences\n)\n)\n:\n# find the end of this pattern\nend_ix\n=\ni\n+\nn\n_\nsteps\n# check if we are beyond the dataset\nif\nend_ix\n>\nlen\n(\nsequences\n)\n:\nbreak\n# gather input and output parts of the pattern\nseq_x\n,\nseq_y\n=\nsequences\n[\ni\n:\nend_ix\n,\n:\n-\n1\n]\n,\nsequences\n[\nend_ix\n-\n1\n,\n-\n1\n]\nX\n.\nappend\n(\nseq_x\n)\ny\n.\nappend\n(\nseq_y\n)\nreturn\narray\n(\nX\n)\n,\narray\n(\ny\n)\nWe can test this function on our dataset using three time steps for each input time series as input.\nThe complete example is listed below.\n# multivariate data preparation\nfrom numpy import array\nfrom numpy import hstack\n\n# split a multivariate sequence into samples\ndef split_sequences(sequences, n_steps):\n\tX, y = list(), list()\n\tfor i in range(len(sequences)):\n\t\t# find the end of this pattern\n\t\tend_ix = i + n_steps\n\t\t# check if we are beyond the dataset\n\t\tif end_ix > len(sequences):\n\t\t\tbreak\n\t\t# gather input and output parts of the pattern\n\t\tseq_x, seq_y = sequences[i:end_ix, :-1], sequences[end_ix-1, -1]\n\t\tX.append(seq_x)\n\t\ty.append(seq_y)\n\treturn array(X), array(y)\n\n# define input sequence\nin_seq1 = array([10, 20, 30, 40, 50, 60, 70, 80, 90])\nin_seq2 = array([15, 25, 35, 45, 55, 65, 75, 85, 95])\nout_seq = array([in_seq1[i]+in_seq2[i] for i in range(len(in_seq1))])\n# convert to [rows, columns] structure\nin_seq1 = in_seq1.reshape((len(in_seq1), 1))\nin_seq2 = in_seq2.reshape((len(in_seq2), 1))\nout_seq = out_seq.reshape((len(out_seq), 1))\n# horizontally stack columns\ndataset = hstack((in_seq1, in_seq2, out_seq))\n# choose a number of time steps\nn_steps = 3\n# convert into input/output\nX, y = split_sequences(dataset, n_steps)\nprint(X.shape, y.shape)\n# summarize the data\nfor i in range(len(X)):\n\tprint(X[i], y[i])\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n14\n15\n16\n17\n18\n19\n20\n21\n22\n23\n24\n25\n26\n27\n28\n29\n30\n31\n32\n33\n34\n35\n36\n37\n# multivariate data preparation\nfrom\nnumpy\nimport\narray\nfrom\nnumpy\nimport\nhstack\n# split a multivariate sequence into samples\ndef\nsplit_sequences\n(\nsequences\n,\nn_steps\n)\n:\nX\n,\ny\n=\nlist\n(\n)\n,\nlist\n(\n)\nfor\ni\nin\nrange\n(\nlen\n(\nsequences\n)\n)\n:\n# find the end of this pattern\nend_ix\n=\ni\n+\nn\n_\nsteps\n# check if we are beyond the dataset\nif\nend_ix\n>\nlen\n(\nsequences\n)\n:\nbreak\n# gather input and output parts of the pattern\nseq_x\n,\nseq_y\n=\nsequences\n[\ni\n:\nend_ix\n,\n:\n-\n1\n]\n,\nsequences\n[\nend_ix\n-\n1\n,\n-\n1\n]\nX\n.\nappend\n(\nseq_x\n)\ny\n.\nappend\n(\nseq_y\n)\nreturn\narray\n(\nX\n)\n,\narray\n(\ny\n)\n# define input sequence\nin_seq1\n=\narray\n(\n[\n10\n,\n20\n,\n30\n,\n40\n,\n50\n,\n60\n,\n70\n,\n80\n,\n90\n]\n)\nin_seq2\n=\narray\n(\n[\n15\n,\n25\n,\n35\n,\n45\n,\n55\n,\n65\n,\n75\n,\n85\n,\n95\n]\n)\nout_seq\n=\narray\n(\n[\nin_seq1\n[\ni\n]\n+\nin_seq2\n[\ni\n]\nfor\ni\nin\nrange\n(\nlen\n(\nin_seq1\n)\n)\n]\n)\n# convert to [rows, columns] structure\nin_seq1\n=\nin_seq1\n.\nreshape\n(\n(\nlen\n(\nin_seq1\n)\n,\n1\n)\n)\nin_seq2\n=\nin_seq2\n.\nreshape\n(\n(\nlen\n(\nin_seq2\n)\n,\n1\n)\n)\nout_seq\n=\nout_seq\n.\nreshape\n(\n(\nlen\n(\nout_seq\n)\n,\n1\n)\n)\n# horizontally stack columns\ndataset\n=\nhstack\n(\n(\nin_seq1\n,\nin_seq2\n,\nout_seq\n)\n)\n# choose a number of time steps\nn_steps\n=\n3\n# convert into input/output\nX\n,\ny\n=\nsplit_sequences\n(\ndataset\n,\nn_steps\n)\nprint\n(\nX\n.\nshape\n,\ny\n.\nshape\n)\n# summarize the data\nfor\ni\nin\nrange\n(\nlen\n(\nX\n)\n)\n:\nprint\n(\nX\n[\ni\n]\n,\ny\n[\ni\n]\n)\nRunning the example first prints the shape of the X and y components.\nWe can see that the X component has a three-dimensional structure.\nThe first dimension is the number of samples, in this case 7. The second dimension is the number of time steps per sample, in this case 3, the value specified to the function. Finally, the last dimension specifies the number of parallel time series or the number of variables, in this case 2 for the two parallel series.\nThis is the exact three-dimensional structure expected by an LSTM as input. The data is ready to use without further reshaping.\nWe can then see that the input and output for each sample is printed, showing the three time steps for each of the two input series and the associated output for each sample.\n(7, 3, 2) (7,)\n\n[[10 15]\n [20 25]\n [30 35]] 65\n[[20 25]\n [30 35]\n [40 45]] 85\n[[30 35]\n [40 45]\n [50 55]] 105\n[[40 45]\n [50 55]\n [60 65]] 125\n[[50 55]\n [60 65]\n [70 75]] 145\n[[60 65]\n [70 75]\n [80 85]] 165\n[[70 75]\n [80 85]\n [90 95]] 185\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n14\n15\n16\n17\n18\n19\n20\n21\n22\n23\n(7, 3, 2) (7,)\n[[10 15]\n[20 25]\n[30 35]] 65\n[[20 25]\n[30 35]\n[40 45]] 85\n[[30 35]\n[40 45]\n[50 55]] 105\n[[40 45]\n[50 55]\n[60 65]] 125\n[[50 55]\n[60 65]\n[70 75]] 145\n[[60 65]\n[70 75]\n[80 85]] 165\n[[70 75]\n[80 85]\n[90 95]] 185\nWe are now ready to fit an LSTM model on this data.\nAny of the varieties of LSTMs in the previous section can be used, such as a Vanilla, Stacked, Bidirectional, CNN, or ConvLSTM model.\nWe will use a Vanilla LSTM where the number of time steps and parallel series (features) are specified for the input layer via the\ninput_shape\nargument.\n...\n# define model\nmodel = Sequential()\nmodel.add(LSTM(50, activation='relu', input_shape=(n_steps, n_features)))\nmodel.add(Dense(1))\nmodel.compile(optimizer='adam', loss='mse')\n1\n2\n3\n4\n5\n6\n.\n.\n.\n# define model\nmodel\n=\nSequential\n(\n)\nmodel\n.\nadd\n(\nLSTM\n(\n50\n,\nactivation\n=\n'relu'\n,\ninput_shape\n=\n(\nn_steps\n,\nn_features\n)\n)\n)\nmodel\n.\nadd\n(\nDense\n(\n1\n)\n)\nmodel\n.\ncompile\n(\noptimizer\n=\n'adam'\n,\nloss\n=\n'mse'\n)\nWhen making a prediction, the model expects three time steps for two input time series.\nWe can predict the next value in the output series providing the input values of:\n80,\t 85\n90,\t 95\n100, 105\n1\n2\n3\n80,\t 85\n90,\t 95\n100, 105\nThe shape of the one sample with three time steps and two variables must be [1, 3, 2].\nWe would expect the next value in the sequence to be 100 + 105, or 205.\n...\n# demonstrate prediction\nx_input = array([[80, 85], [90, 95], [100, 105]])\nx_input = x_input.reshape((1, n_steps, n_features))\nyhat = model.predict(x_input, verbose=0)\n1\n2\n3\n4\n5\n.\n.\n.\n# demonstrate prediction\nx_input\n=\narray\n(\n[\n[\n80\n,\n85\n]\n,\n[\n90\n,\n95\n]\n,\n[\n100\n,\n105\n]\n]\n)\nx_input\n=\nx_input\n.\nreshape\n(\n(\n1\n,\nn_steps\n,\nn_features\n)\n)\nyhat\n=\nmodel\n.\npredict\n(\nx_input\n,\nverbose\n=\n0\n)\nThe complete example is listed below.\n# multivariate lstm example\nfrom numpy import array\nfrom numpy import hstack\nfrom keras.models import Sequential\nfrom keras.layers import LSTM\nfrom keras.layers import Dense\n\n# split a multivariate sequence into samples\ndef split_sequences(sequences, n_steps):\n\tX, y = list(), list()\n\tfor i in range(len(sequences)):\n\t\t# find the end of this pattern\n\t\tend_ix = i + n_steps\n\t\t# check if we are beyond the dataset\n\t\tif end_ix > len(sequences):\n\t\t\tbreak\n\t\t# gather input and output parts of the pattern\n\t\tseq_x, seq_y = sequences[i:end_ix, :-1], sequences[end_ix-1, -1]\n\t\tX.append(seq_x)\n\t\ty.append(seq_y)\n\treturn array(X), array(y)\n\n# define input sequence\nin_seq1 = array([10, 20, 30, 40, 50, 60, 70, 80, 90])\nin_seq2 = array([15, 25, 35, 45, 55, 65, 75, 85, 95])\nout_seq = array([in_seq1[i]+in_seq2[i] for i in range(len(in_seq1))])\n# convert to [rows, columns] structure\nin_seq1 = in_seq1.reshape((len(in_seq1), 1))\nin_seq2 = in_seq2.reshape((len(in_seq2), 1))\nout_seq = out_seq.reshape((len(out_seq), 1))\n# horizontally stack columns\ndataset = hstack((in_seq1, in_seq2, out_seq))\n# choose a number of time steps\nn_steps = 3\n# convert into input/output\nX, y = split_sequences(dataset, n_steps)\n# the dataset knows the number of features, e.g. 2\nn_features = X.shape[2]\n# define model\nmodel = Sequential()\nmodel.add(LSTM(50, activation='relu', input_shape=(n_steps, n_features)))\nmodel.add(Dense(1))\nmodel.compile(optimizer='adam', loss='mse')\n# fit model\nmodel.fit(X, y, epochs=200, verbose=0)\n# demonstrate prediction\nx_input = array([[80, 85], [90, 95], [100, 105]])\nx_input = x_input.reshape((1, n_steps, n_features))\nyhat = model.predict(x_input, verbose=0)\nprint(yhat)\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n14\n15\n16\n17\n18\n19\n20\n21\n22\n23\n24\n25\n26\n27\n28\n29\n30\n31\n32\n33\n34\n35\n36\n37\n38\n39\n40\n41\n42\n43\n44\n45\n46\n47\n48\n49\n50\n# multivariate lstm example\nfrom\nnumpy\nimport\narray\nfrom\nnumpy\nimport\nhstack\nfrom\nkeras\n.\nmodels\nimport\nSequential\nfrom\nkeras\n.\nlayers\nimport\nLSTM\nfrom\nkeras\n.\nlayers\nimport\nDense\n# split a multivariate sequence into samples\ndef\nsplit_sequences\n(\nsequences\n,\nn_steps\n)\n:\nX\n,\ny\n=\nlist\n(\n)\n,\nlist\n(\n)\nfor\ni\nin\nrange\n(\nlen\n(\nsequences\n)\n)\n:\n# find the end of this pattern\nend_ix\n=\ni\n+\nn\n_\nsteps\n# check if we are beyond the dataset\nif\nend_ix\n>\nlen\n(\nsequences\n)\n:\nbreak\n# gather input and output parts of the pattern\nseq_x\n,\nseq_y\n=\nsequences\n[\ni\n:\nend_ix\n,\n:\n-\n1\n]\n,\nsequences\n[\nend_ix\n-\n1\n,\n-\n1\n]\nX\n.\nappend\n(\nseq_x\n)\ny\n.\nappend\n(\nseq_y\n)\nreturn\narray\n(\nX\n)\n,\narray\n(\ny\n)\n# define input sequence\nin_seq1\n=\narray\n(\n[\n10\n,\n20\n,\n30\n,\n40\n,\n50\n,\n60\n,\n70\n,\n80\n,\n90\n]\n)\nin_seq2\n=\narray\n(\n[\n15\n,\n25\n,\n35\n,\n45\n,\n55\n,\n65\n,\n75\n,\n85\n,\n95\n]\n)\nout_seq\n=\narray\n(\n[\nin_seq1\n[\ni\n]\n+\nin_seq2\n[\ni\n]\nfor\ni\nin\nrange\n(\nlen\n(\nin_seq1\n)\n)\n]\n)\n# convert to [rows, columns] structure\nin_seq1\n=\nin_seq1\n.\nreshape\n(\n(\nlen\n(\nin_seq1\n)\n,\n1\n)\n)\nin_seq2\n=\nin_seq2\n.\nreshape\n(\n(\nlen\n(\nin_seq2\n)\n,\n1\n)\n)\nout_seq\n=\nout_seq\n.\nreshape\n(\n(\nlen\n(\nout_seq\n)\n,\n1\n)\n)\n# horizontally stack columns\ndataset\n=\nhstack\n(\n(\nin_seq1\n,\nin_seq2\n,\nout_seq\n)\n)\n# choose a number of time steps\nn_steps\n=\n3\n# convert into input/output\nX\n,\ny\n=\nsplit_sequences\n(\ndataset\n,\nn_steps\n)\n# the dataset knows the number of features, e.g. 2\nn_features\n=\nX\n.\nshape\n[\n2\n]\n# define model\nmodel\n=\nSequential\n(\n)\nmodel\n.\nadd\n(\nLSTM\n(\n50\n,\nactivation\n=\n'relu'\n,\ninput_shape\n=\n(\nn_steps\n,\nn_features\n)\n)\n)\nmodel\n.\nadd\n(\nDense\n(\n1\n)\n)\nmodel\n.\ncompile\n(\noptimizer\n=\n'adam'\n,\nloss\n=\n'mse'\n)\n# fit model\nmodel\n.\nfit\n(\nX\n,\ny\n,\nepochs\n=\n200\n,\nverbose\n=\n0\n)\n# demonstrate prediction\nx_input\n=\narray\n(\n[\n[\n80\n,\n85\n]\n,\n[\n90\n,\n95\n]\n,\n[\n100\n,\n105\n]\n]\n)\nx_input\n=\nx_input\n.\nreshape\n(\n(\n1\n,\nn_steps\n,\nn_features\n)\n)\nyhat\n=\nmodel\n.\npredict\n(\nx_input\n,\nverbose\n=\n0\n)\nprint\n(\nyhat\n)\nNote\n: Your\nresults may vary\ngiven the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.\nRunning the example prepares the data, fits the model, and makes a prediction.\n[[208.13531]]\n1\n[[208.13531]]\nMultiple Parallel Series\nAn alternate time series problem is the case where there are multiple parallel time series and a value must be predicted for each.\nFor example, given the data from the previous section:\n[[ 10  15  25]\n [ 20  25  45]\n [ 30  35  65]\n [ 40  45  85]\n [ 50  55 105]\n [ 60  65 125]\n [ 70  75 145]\n [ 80  85 165]\n [ 90  95 185]]\n1\n2\n3\n4\n5\n6\n7\n8\n9\n[[ 10  15  25]\n[ 20  25  45]\n[ 30  35  65]\n[ 40  45  85]\n[ 50  55 105]\n[ 60  65 125]\n[ 70  75 145]\n[ 80  85 165]\n[ 90  95 185]]\nWe may want to predict the value for each of the three time series for the next time step.\nThis might be referred to as multivariate forecasting.\nAgain, the data must be split into input/output samples in order to train a model.\nThe first sample of this dataset would be:\nInput:\n10, 15, 25\n20, 25, 45\n30, 35, 65\n1\n2\n3\n10, 15, 25\n20, 25, 45\n30, 35, 65\nOutput:\n40, 45, 85\n1\n40, 45, 85\nThe\nsplit_sequences()\nfunction below will split multiple parallel time series with rows for time steps and one series per column into the required input/output shape.\n# split a multivariate sequence into samples\ndef split_sequences(sequences, n_steps):\n\tX, y = list(), list()\n\tfor i in range(len(sequences)):\n\t\t# find the end of this pattern\n\t\tend_ix = i + n_steps\n\t\t# check if we are beyond the dataset\n\t\tif end_ix > len(sequences)-1:\n\t\t\tbreak\n\t\t# gather input and output parts of the pattern\n\t\tseq_x, seq_y = sequences[i:end_ix, :], sequences[end_ix, :]\n\t\tX.append(seq_x)\n\t\ty.append(seq_y)\n\treturn array(X), array(y)\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n14\n# split a multivariate sequence into samples\ndef\nsplit_sequences\n(\nsequences\n,\nn_steps\n)\n:\nX\n,\ny\n=\nlist\n(\n)\n,\nlist\n(\n)\nfor\ni\nin\nrange\n(\nlen\n(\nsequences\n)\n)\n:\n# find the end of this pattern\nend_ix\n=\ni\n+\nn\n_\nsteps\n# check if we are beyond the dataset\nif\nend_ix\n>\nlen\n(\nsequences\n)\n-\n1\n:\nbreak\n# gather input and output parts of the pattern\nseq_x\n,\nseq_y\n=\nsequences\n[\ni\n:\nend_ix\n,\n:\n]\n,\nsequences\n[\nend_ix\n,\n:\n]\nX\n.\nappend\n(\nseq_x\n)\ny\n.\nappend\n(\nseq_y\n)\nreturn\narray\n(\nX\n)\n,\narray\n(\ny\n)\nWe can demonstrate this on the contrived problem; the complete example is listed below.\n# multivariate output data prep\nfrom numpy import array\nfrom numpy import hstack\n\n# split a multivariate sequence into samples\ndef split_sequences(sequences, n_steps):\n\tX, y = list(), list()\n\tfor i in range(len(sequences)):\n\t\t# find the end of this pattern\n\t\tend_ix = i + n_steps\n\t\t# check if we are beyond the dataset\n\t\tif end_ix > len(sequences)-1:\n\t\t\tbreak\n\t\t# gather input and output parts of the pattern\n\t\tseq_x, seq_y = sequences[i:end_ix, :], sequences[end_ix, :]\n\t\tX.append(seq_x)\n\t\ty.append(seq_y)\n\treturn array(X), array(y)\n\n# define input sequence\nin_seq1 = array([10, 20, 30, 40, 50, 60, 70, 80, 90])\nin_seq2 = array([15, 25, 35, 45, 55, 65, 75, 85, 95])\nout_seq = array([in_seq1[i]+in_seq2[i] for i in range(len(in_seq1))])\n# convert to [rows, columns] structure\nin_seq1 = in_seq1.reshape((len(in_seq1), 1))\nin_seq2 = in_seq2.reshape((len(in_seq2), 1))\nout_seq = out_seq.reshape((len(out_seq), 1))\n# horizontally stack columns\ndataset = hstack((in_seq1, in_seq2, out_seq))\n# choose a number of time steps\nn_steps = 3\n# convert into input/output\nX, y = split_sequences(dataset, n_steps)\nprint(X.shape, y.shape)\n# summarize the data\nfor i in range(len(X)):\n\tprint(X[i], y[i])\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n14\n15\n16\n17\n18\n19\n20\n21\n22\n23\n24\n25\n26\n27\n28\n29\n30\n31\n32\n33\n34\n35\n36\n37\n# multivariate output data prep\nfrom\nnumpy\nimport\narray\nfrom\nnumpy\nimport\nhstack\n# split a multivariate sequence into samples\ndef\nsplit_sequences\n(\nsequences\n,\nn_steps\n)\n:\nX\n,\ny\n=\nlist\n(\n)\n,\nlist\n(\n)\nfor\ni\nin\nrange\n(\nlen\n(\nsequences\n)\n)\n:\n# find the end of this pattern\nend_ix\n=\ni\n+\nn\n_\nsteps\n# check if we are beyond the dataset\nif\nend_ix\n>\nlen\n(\nsequences\n)\n-\n1\n:\nbreak\n# gather input and output parts of the pattern\nseq_x\n,\nseq_y\n=\nsequences\n[\ni\n:\nend_ix\n,\n:\n]\n,\nsequences\n[\nend_ix\n,\n:\n]\nX\n.\nappend\n(\nseq_x\n)\ny\n.\nappend\n(\nseq_y\n)\nreturn\narray\n(\nX\n)\n,\narray\n(\ny\n)\n# define input sequence\nin_seq1\n=\narray\n(\n[\n10\n,\n20\n,\n30\n,\n40\n,\n50\n,\n60\n,\n70\n,\n80\n,\n90\n]\n)\nin_seq2\n=\narray\n(\n[\n15\n,\n25\n,\n35\n,\n45\n,\n55\n,\n65\n,\n75\n,\n85\n,\n95\n]\n)\nout_seq\n=\narray\n(\n[\nin_seq1\n[\ni\n]\n+\nin_seq2\n[\ni\n]\nfor\ni\nin\nrange\n(\nlen\n(\nin_seq1\n)\n)\n]\n)\n# convert to [rows, columns] structure\nin_seq1\n=\nin_seq1\n.\nreshape\n(\n(\nlen\n(\nin_seq1\n)\n,\n1\n)\n)\nin_seq2\n=\nin_seq2\n.\nreshape\n(\n(\nlen\n(\nin_seq2\n)\n,\n1\n)\n)\nout_seq\n=\nout_seq\n.\nreshape\n(\n(\nlen\n(\nout_seq\n)\n,\n1\n)\n)\n# horizontally stack columns\ndataset\n=\nhstack\n(\n(\nin_seq1\n,\nin_seq2\n,\nout_seq\n)\n)\n# choose a number of time steps\nn_steps\n=\n3\n# convert into input/output\nX\n,\ny\n=\nsplit_sequences\n(\ndataset\n,\nn_steps\n)\nprint\n(\nX\n.\nshape\n,\ny\n.\nshape\n)\n# summarize the data\nfor\ni\nin\nrange\n(\nlen\n(\nX\n)\n)\n:\nprint\n(\nX\n[\ni\n]\n,\ny\n[\ni\n]\n)\nRunning the example first prints the shape of the prepared X and y components.\nThe shape of X is three-dimensional, including the number of samples (6), the number of time steps chosen per sample (3), and the number of parallel time series or features (3).\nThe shape of y is two-dimensional as we might expect for the number of samples (6) and the number of time variables per sample to be predicted (3).\nThe data is ready to use in an LSTM model that expects three-dimensional input and two-dimensional output shapes for the X and y components of each sample.\nThen, each of the samples is printed showing the input and output components of each sample.\n(6, 3, 3) (6, 3)\n\n[[10 15 25]\n [20 25 45]\n [30 35 65]] [40 45 85]\n[[20 25 45]\n [30 35 65]\n [40 45 85]] [ 50  55 105]\n[[ 30  35  65]\n [ 40  45  85]\n [ 50  55 105]] [ 60  65 125]\n[[ 40  45  85]\n [ 50  55 105]\n [ 60  65 125]] [ 70  75 145]\n[[ 50  55 105]\n [ 60  65 125]\n [ 70  75 145]] [ 80  85 165]\n[[ 60  65 125]\n [ 70  75 145]\n [ 80  85 165]] [ 90  95 185]\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n14\n15\n16\n17\n18\n19\n20\n(6, 3, 3) (6, 3)\n[[10 15 25]\n[20 25 45]\n[30 35 65]] [40 45 85]\n[[20 25 45]\n[30 35 65]\n[40 45 85]] [ 50  55 105]\n[[ 30  35  65]\n[ 40  45  85]\n[ 50  55 105]] [ 60  65 125]\n[[ 40  45  85]\n[ 50  55 105]\n[ 60  65 125]] [ 70  75 145]\n[[ 50  55 105]\n[ 60  65 125]\n[ 70  75 145]] [ 80  85 165]\n[[ 60  65 125]\n[ 70  75 145]\n[ 80  85 165]] [ 90  95 185]\nWe are now ready to fit an LSTM model on this data.\nAny of the varieties of LSTMs in the previous section can be used, such as a Vanilla, Stacked, Bidirectional, CNN, or ConvLSTM model.\nWe will use a Stacked LSTM where the number of time steps and parallel series (features) are specified for the input layer via the\ninput_shape\nargument. The number of parallel series is also used in the specification of the number of values to predict by the model in the output layer; again, this is three.\n...\n# define model\nmodel = Sequential()\nmodel.add(LSTM(100, activation='relu', return_sequences=True, input_shape=(n_steps, n_features)))\nmodel.add(LSTM(100, activation='relu'))\nmodel.add(Dense(n_features))\nmodel.compile(optimizer='adam', loss='mse')\n1\n2\n3\n4\n5\n6\n7\n.\n.\n.\n# define model\nmodel\n=\nSequential\n(\n)\nmodel\n.\nadd\n(\nLSTM\n(\n100\n,\nactivation\n=\n'relu'\n,\nreturn_sequences\n=\nTrue\n,\ninput_shape\n=\n(\nn_steps\n,\nn_features\n)\n)\n)\nmodel\n.\nadd\n(\nLSTM\n(\n100\n,\nactivation\n=\n'relu'\n)\n)\nmodel\n.\nadd\n(\nDense\n(\nn_features\n)\n)\nmodel\n.\ncompile\n(\noptimizer\n=\n'adam'\n,\nloss\n=\n'mse'\n)\nWe can predict the next value in each of the three parallel series by providing an input of three time steps for each series.\n70, 75, 145\n80, 85, 165\n90, 95, 185\n1\n2\n3\n70, 75, 145\n80, 85, 165\n90, 95, 185\nThe shape of the input for making a single prediction must be 1 sample, 3 time steps, and 3 features, or [1, 3, 3]\n...\n# demonstrate prediction\nx_input = array([[70,75,145], [80,85,165], [90,95,185]])\nx_input = x_input.reshape((1, n_steps, n_features))\nyhat = model.predict(x_input, verbose=0)\n1\n2\n3\n4\n5\n.\n.\n.\n# demonstrate prediction\nx_input\n=\narray\n(\n[\n[\n70\n,\n75\n,\n145\n]\n,\n[\n80\n,\n85\n,\n165\n]\n,\n[\n90\n,\n95\n,\n185\n]\n]\n)\nx_input\n=\nx_input\n.\nreshape\n(\n(\n1\n,\nn_steps\n,\nn_features\n)\n)\nyhat\n=\nmodel\n.\npredict\n(\nx_input\n,\nverbose\n=\n0\n)\nWe would expect the vector output to be:\n[100, 105, 205]\n1\n[100, 105, 205]\nWe can tie all of this together and demonstrate a Stacked LSTM for multivariate output time series forecasting below.\n# multivariate output stacked lstm example\nfrom numpy import array\nfrom numpy import hstack\nfrom keras.models import Sequential\nfrom keras.layers import LSTM\nfrom keras.layers import Dense\n\n# split a multivariate sequence into samples\ndef split_sequences(sequences, n_steps):\n\tX, y = list(), list()\n\tfor i in range(len(sequences)):\n\t\t# find the end of this pattern\n\t\tend_ix = i + n_steps\n\t\t# check if we are beyond the dataset\n\t\tif end_ix > len(sequences)-1:\n\t\t\tbreak\n\t\t# gather input and output parts of the pattern\n\t\tseq_x, seq_y = sequences[i:end_ix, :], sequences[end_ix, :]\n\t\tX.append(seq_x)\n\t\ty.append(seq_y)\n\treturn array(X), array(y)\n\n# define input sequence\nin_seq1 = array([10, 20, 30, 40, 50, 60, 70, 80, 90])\nin_seq2 = array([15, 25, 35, 45, 55, 65, 75, 85, 95])\nout_seq = array([in_seq1[i]+in_seq2[i] for i in range(len(in_seq1))])\n# convert to [rows, columns] structure\nin_seq1 = in_seq1.reshape((len(in_seq1), 1))\nin_seq2 = in_seq2.reshape((len(in_seq2), 1))\nout_seq = out_seq.reshape((len(out_seq), 1))\n# horizontally stack columns\ndataset = hstack((in_seq1, in_seq2, out_seq))\n# choose a number of time steps\nn_steps = 3\n# convert into input/output\nX, y = split_sequences(dataset, n_steps)\n# the dataset knows the number of features, e.g. 2\nn_features = X.shape[2]\n# define model\nmodel = Sequential()\nmodel.add(LSTM(100, activation='relu', return_sequences=True, input_shape=(n_steps, n_features)))\nmodel.add(LSTM(100, activation='relu'))\nmodel.add(Dense(n_features))\nmodel.compile(optimizer='adam', loss='mse')\n# fit model\nmodel.fit(X, y, epochs=400, verbose=0)\n# demonstrate prediction\nx_input = array([[70,75,145], [80,85,165], [90,95,185]])\nx_input = x_input.reshape((1, n_steps, n_features))\nyhat = model.predict(x_input, verbose=0)\nprint(yhat)\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n14\n15\n16\n17\n18\n19\n20\n21\n22\n23\n24\n25\n26\n27\n28\n29\n30\n31\n32\n33\n34\n35\n36\n37\n38\n39\n40\n41\n42\n43\n44\n45\n46\n47\n48\n49\n50\n51\n# multivariate output stacked lstm example\nfrom\nnumpy\nimport\narray\nfrom\nnumpy\nimport\nhstack\nfrom\nkeras\n.\nmodels\nimport\nSequential\nfrom\nkeras\n.\nlayers\nimport\nLSTM\nfrom\nkeras\n.\nlayers\nimport\nDense\n# split a multivariate sequence into samples\ndef\nsplit_sequences\n(\nsequences\n,\nn_steps\n)\n:\nX\n,\ny\n=\nlist\n(\n)\n,\nlist\n(\n)\nfor\ni\nin\nrange\n(\nlen\n(\nsequences\n)\n)\n:\n# find the end of this pattern\nend_ix\n=\ni\n+\nn\n_\nsteps\n# check if we are beyond the dataset\nif\nend_ix\n>\nlen\n(\nsequences\n)\n-\n1\n:\nbreak\n# gather input and output parts of the pattern\nseq_x\n,\nseq_y\n=\nsequences\n[\ni\n:\nend_ix\n,\n:\n]\n,\nsequences\n[\nend_ix\n,\n:\n]\nX\n.\nappend\n(\nseq_x\n)\ny\n.\nappend\n(\nseq_y\n)\nreturn\narray\n(\nX\n)\n,\narray\n(\ny\n)\n# define input sequence\nin_seq1\n=\narray\n(\n[\n10\n,\n20\n,\n30\n,\n40\n,\n50\n,\n60\n,\n70\n,\n80\n,\n90\n]\n)\nin_seq2\n=\narray\n(\n[\n15\n,\n25\n,\n35\n,\n45\n,\n55\n,\n65\n,\n75\n,\n85\n,\n95\n]\n)\nout_seq\n=\narray\n(\n[\nin_seq1\n[\ni\n]\n+\nin_seq2\n[\ni\n]\nfor\ni\nin\nrange\n(\nlen\n(\nin_seq1\n)\n)\n]\n)\n# convert to [rows, columns] structure\nin_seq1\n=\nin_seq1\n.\nreshape\n(\n(\nlen\n(\nin_seq1\n)\n,\n1\n)\n)\nin_seq2\n=\nin_seq2\n.\nreshape\n(\n(\nlen\n(\nin_seq2\n)\n,\n1\n)\n)\nout_seq\n=\nout_seq\n.\nreshape\n(\n(\nlen\n(\nout_seq\n)\n,\n1\n)\n)\n# horizontally stack columns\ndataset\n=\nhstack\n(\n(\nin_seq1\n,\nin_seq2\n,\nout_seq\n)\n)\n# choose a number of time steps\nn_steps\n=\n3\n# convert into input/output\nX\n,\ny\n=\nsplit_sequences\n(\ndataset\n,\nn_steps\n)\n# the dataset knows the number of features, e.g. 2\nn_features\n=\nX\n.\nshape\n[\n2\n]\n# define model\nmodel\n=\nSequential\n(\n)\nmodel\n.\nadd\n(\nLSTM\n(\n100\n,\nactivation\n=\n'relu'\n,\nreturn_sequences\n=\nTrue\n,\ninput_shape\n=\n(\nn_steps\n,\nn_features\n)\n)\n)\nmodel\n.\nadd\n(\nLSTM\n(\n100\n,\nactivation\n=\n'relu'\n)\n)\nmodel\n.\nadd\n(\nDense\n(\nn_features\n)\n)\nmodel\n.\ncompile\n(\noptimizer\n=\n'adam'\n,\nloss\n=\n'mse'\n)\n# fit model\nmodel\n.\nfit\n(\nX\n,\ny\n,\nepochs\n=\n400\n,\nverbose\n=\n0\n)\n# demonstrate prediction\nx_input\n=\narray\n(\n[\n[\n70\n,\n75\n,\n145\n]\n,\n[\n80\n,\n85\n,\n165\n]\n,\n[\n90\n,\n95\n,\n185\n]\n]\n)\nx_input\n=\nx_input\n.\nreshape\n(\n(\n1\n,\nn_steps\n,\nn_features\n)\n)\nyhat\n=\nmodel\n.\npredict\n(\nx_input\n,\nverbose\n=\n0\n)\nprint\n(\nyhat\n)\nNote\n: Your\nresults may vary\ngiven the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.\nRunning the example prepares the data, fits the model, and makes a prediction.\n[[101.76599 108.730484 206.63577 ]]\n1\n[[101.76599 108.730484 206.63577 ]]\nMulti-Step LSTM Models\nA time series forecasting problem that requires a prediction of multiple time steps into the future can be referred to as multi-step time series forecasting.\nSpecifically, these are problems where the forecast horizon or interval is more than one time step.\nThere are two main types of LSTM models that can be used for multi-step forecasting; they are:\nVector Output Model\nEncoder-Decoder Model\nBefore we look at these models, let’s first look at the preparation of data for multi-step forecasting.\nData Preparation\nAs with one-step forecasting, a time series used for multi-step time series forecasting must be split into samples with input and output components.\nBoth the input and output components will be comprised of multiple time steps and may or may not have the same number of steps.\nFor example, given the univariate time series:\n[10, 20, 30, 40, 50, 60, 70, 80, 90]\n1\n[10, 20, 30, 40, 50, 60, 70, 80, 90]\nWe could use the last three time steps as input and forecast the next two time steps.\nThe first sample would look as follows:\nInput:\n[10, 20, 30]\n1\n[10, 20, 30]\nOutput:\n[40, 50]\n1\n[40, 50]\nThe\nsplit_sequence()\nfunction below implements this behavior and will split a given univariate time series into samples with a specified number of input and output time steps.\n# split a univariate sequence into samples\ndef split_sequence(sequence, n_steps_in, n_steps_out):\n\tX, y = list(), list()\n\tfor i in range(len(sequence)):\n\t\t# find the end of this pattern\n\t\tend_ix = i + n_steps_in\n\t\tout_end_ix = end_ix + n_steps_out\n\t\t# check if we are beyond the sequence\n\t\tif out_end_ix > len(sequence):\n\t\t\tbreak\n\t\t# gather input and output parts of the pattern\n\t\tseq_x, seq_y = sequence[i:end_ix], sequence[end_ix:out_end_ix]\n\t\tX.append(seq_x)\n\t\ty.append(seq_y)\n\treturn array(X), array(y)\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n14\n15\n# split a univariate sequence into samples\ndef\nsplit_sequence\n(\nsequence\n,\nn_steps_in\n,\nn_steps_out\n)\n:\nX\n,\ny\n=\nlist\n(\n)\n,\nlist\n(\n)\nfor\ni\nin\nrange\n(\nlen\n(\nsequence\n)\n)\n:\n# find the end of this pattern\nend_ix\n=\ni\n+\nn_steps_in\nout_end_ix\n=\nend_ix\n+\nn_steps\n_\nout\n# check if we are beyond the sequence\nif\nout_end_ix\n>\nlen\n(\nsequence\n)\n:\nbreak\n# gather input and output parts of the pattern\nseq_x\n,\nseq_y\n=\nsequence\n[\ni\n:\nend_ix\n]\n,\nsequence\n[\nend_ix\n:\nout_end_ix\n]\nX\n.\nappend\n(\nseq_x\n)\ny\n.\nappend\n(\nseq_y\n)\nreturn\narray\n(\nX\n)\n,\narray\n(\ny\n)\nWe can demonstrate this function on the small contrived dataset.\nThe complete example is listed below.\n# multi-step data preparation\nfrom numpy import array\n\n# split a univariate sequence into samples\ndef split_sequence(sequence, n_steps_in, n_steps_out):\n\tX, y = list(), list()\n\tfor i in range(len(sequence)):\n\t\t# find the end of this pattern\n\t\tend_ix = i + n_steps_in\n\t\tout_end_ix = end_ix + n_steps_out\n\t\t# check if we are beyond the sequence\n\t\tif out_end_ix > len(sequence):\n\t\t\tbreak\n\t\t# gather input and output parts of the pattern\n\t\tseq_x, seq_y = sequence[i:end_ix], sequence[end_ix:out_end_ix]\n\t\tX.append(seq_x)\n\t\ty.append(seq_y)\n\treturn array(X), array(y)\n\n# define input sequence\nraw_seq = [10, 20, 30, 40, 50, 60, 70, 80, 90]\n# choose a number of time steps\nn_steps_in, n_steps_out = 3, 2\n# split into samples\nX, y = split_sequence(raw_seq, n_steps_in, n_steps_out)\n# summarize the data\nfor i in range(len(X)):\n\tprint(X[i], y[i])\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n14\n15\n16\n17\n18\n19\n20\n21\n22\n23\n24\n25\n26\n27\n28\n# multi-step data preparation\nfrom\nnumpy\nimport\narray\n# split a univariate sequence into samples\ndef\nsplit_sequence\n(\nsequence\n,\nn_steps_in\n,\nn_steps_out\n)\n:\nX\n,\ny\n=\nlist\n(\n)\n,\nlist\n(\n)\nfor\ni\nin\nrange\n(\nlen\n(\nsequence\n)\n)\n:\n# find the end of this pattern\nend_ix\n=\ni\n+\nn_steps_in\nout_end_ix\n=\nend_ix\n+\nn_steps\n_\nout\n# check if we are beyond the sequence\nif\nout_end_ix\n>\nlen\n(\nsequence\n)\n:\nbreak\n# gather input and output parts of the pattern\nseq_x\n,\nseq_y\n=\nsequence\n[\ni\n:\nend_ix\n]\n,\nsequence\n[\nend_ix\n:\nout_end_ix\n]\nX\n.\nappend\n(\nseq_x\n)\ny\n.\nappend\n(\nseq_y\n)\nreturn\narray\n(\nX\n)\n,\narray\n(\ny\n)\n# define input sequence\nraw_seq\n=\n[\n10\n,\n20\n,\n30\n,\n40\n,\n50\n,\n60\n,\n70\n,\n80\n,\n90\n]\n# choose a number of time steps\nn_steps_in\n,\nn_steps_out\n=\n3\n,\n2\n# split into samples\nX\n,\ny\n=\nsplit_sequence\n(\nraw_seq\n,\nn_steps_in\n,\nn_steps_out\n)\n# summarize the data\nfor\ni\nin\nrange\n(\nlen\n(\nX\n)\n)\n:\nprint\n(\nX\n[\ni\n]\n,\ny\n[\ni\n]\n)\nRunning the example splits the univariate series into input and output time steps and prints the input and output components of each.\n[10 20 30] [40 50]\n[20 30 40] [50 60]\n[30 40 50] [60 70]\n[40 50 60] [70 80]\n[50 60 70] [80 90]\n1\n2\n3\n4\n5\n[10 20 30] [40 50]\n[20 30 40] [50 60]\n[30 40 50] [60 70]\n[40 50 60] [70 80]\n[50 60 70] [80 90]\nNow that we know how to prepare data for multi-step forecasting, let’s look at some LSTM models that can learn this mapping.\nVector Output Model\nLike other types of neural network models, the LSTM can output a vector directly that can be interpreted as a multi-step forecast.\nThis approach was seen in the previous section were one time step of each output time series was forecasted as a vector.\nAs with the LSTMs for univariate data in a prior section, the prepared samples must first be reshaped. The LSTM expects data to have a three-dimensional structure of [\nsamples, timesteps, features\n], and in this case, we only have one feature so the reshape is straightforward.\n...\n# reshape from [samples, timesteps] into [samples, timesteps, features]\nn_features = 1\nX = X.reshape((X.shape[0], X.shape[1], n_features))\n1\n2\n3\n4\n.\n.\n.\n# reshape from [samples, timesteps] into [samples, timesteps, features]\nn_features\n=\n1\nX\n=\nX\n.\nreshape\n(\n(\nX\n.\nshape\n[\n0\n]\n,\nX\n.\nshape\n[\n1\n]\n,\nn_features\n)\n)\nWith the number of input and output steps specified in the\nn_steps_in\nand\nn_steps_out\nvariables, we can define a multi-step time-series forecasting model.\nAny of the presented LSTM model types could be used, such as Vanilla, Stacked, Bidirectional, CNN-LSTM, or ConvLSTM. Below defines a Stacked LSTM for multi-step forecasting.\n...\n# define model\nmodel = Sequential()\nmodel.add(LSTM(100, activation='relu', return_sequences=True, input_shape=(n_steps_in, n_features)))\nmodel.add(LSTM(100, activation='relu'))\nmodel.add(Dense(n_steps_out))\nmodel.compile(optimizer='adam', loss='mse')\n1\n2\n3\n4\n5\n6\n7\n.\n.\n.\n# define model\nmodel\n=\nSequential\n(\n)\nmodel\n.\nadd\n(\nLSTM\n(\n100\n,\nactivation\n=\n'relu'\n,\nreturn_sequences\n=\nTrue\n,\ninput_shape\n=\n(\nn_steps_in\n,\nn_features\n)\n)\n)\nmodel\n.\nadd\n(\nLSTM\n(\n100\n,\nactivation\n=\n'relu'\n)\n)\nmodel\n.\nadd\n(\nDense\n(\nn_steps_out\n)\n)\nmodel\n.\ncompile\n(\noptimizer\n=\n'adam'\n,\nloss\n=\n'mse'\n)\nThe model can make a prediction for a single sample. We can predict the next two steps beyond the end of the dataset by providing the input:\n[70, 80, 90]\n1\n[70, 80, 90]\nWe would expect the predicted output to be:\n[100, 110]\n1\n[100, 110]\nAs expected by the model, the shape of the single sample of input data when making the prediction must be [1, 3, 1] for the 1 sample, 3 time steps of the input, and the single feature.\n...\n# demonstrate prediction\nx_input = array([70, 80, 90])\nx_input = x_input.reshape((1, n_steps_in, n_features))\nyhat = model.predict(x_input, verbose=0)\n1\n2\n3\n4\n5\n.\n.\n.\n# demonstrate prediction\nx_input\n=\narray\n(\n[\n70\n,\n80\n,\n90\n]\n)\nx_input\n=\nx_input\n.\nreshape\n(\n(\n1\n,\nn_steps_in\n,\nn_features\n)\n)\nyhat\n=\nmodel\n.\npredict\n(\nx_input\n,\nverbose\n=\n0\n)\nTying all of this together, the Stacked LSTM for multi-step forecasting with a univariate time series is listed below.\n# univariate multi-step vector-output stacked lstm example\nfrom numpy import array\nfrom keras.models import Sequential\nfrom keras.layers import LSTM\nfrom keras.layers import Dense\n\n# split a univariate sequence into samples\ndef split_sequence(sequence, n_steps_in, n_steps_out):\n\tX, y = list(), list()\n\tfor i in range(len(sequence)):\n\t\t# find the end of this pattern\n\t\tend_ix = i + n_steps_in\n\t\tout_end_ix = end_ix + n_steps_out\n\t\t# check if we are beyond the sequence\n\t\tif out_end_ix > len(sequence):\n\t\t\tbreak\n\t\t# gather input and output parts of the pattern\n\t\tseq_x, seq_y = sequence[i:end_ix], sequence[end_ix:out_end_ix]\n\t\tX.append(seq_x)\n\t\ty.append(seq_y)\n\treturn array(X), array(y)\n\n# define input sequence\nraw_seq = [10, 20, 30, 40, 50, 60, 70, 80, 90]\n# choose a number of time steps\nn_steps_in, n_steps_out = 3, 2\n# split into samples\nX, y = split_sequence(raw_seq, n_steps_in, n_steps_out)\n# reshape from [samples, timesteps] into [samples, timesteps, features]\nn_features = 1\nX = X.reshape((X.shape[0], X.shape[1], n_features))\n# define model\nmodel = Sequential()\nmodel.add(LSTM(100, activation='relu', return_sequences=True, input_shape=(n_steps_in, n_features)))\nmodel.add(LSTM(100, activation='relu'))\nmodel.add(Dense(n_steps_out))\nmodel.compile(optimizer='adam', loss='mse')\n# fit model\nmodel.fit(X, y, epochs=50, verbose=0)\n# demonstrate prediction\nx_input = array([70, 80, 90])\nx_input = x_input.reshape((1, n_steps_in, n_features))\nyhat = model.predict(x_input, verbose=0)\nprint(yhat)\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n14\n15\n16\n17\n18\n19\n20\n21\n22\n23\n24\n25\n26\n27\n28\n29\n30\n31\n32\n33\n34\n35\n36\n37\n38\n39\n40\n41\n42\n43\n44\n# univariate multi-step vector-output stacked lstm example\nfrom\nnumpy\nimport\narray\nfrom\nkeras\n.\nmodels\nimport\nSequential\nfrom\nkeras\n.\nlayers\nimport\nLSTM\nfrom\nkeras\n.\nlayers\nimport\nDense\n# split a univariate sequence into samples\ndef\nsplit_sequence\n(\nsequence\n,\nn_steps_in\n,\nn_steps_out\n)\n:\nX\n,\ny\n=\nlist\n(\n)\n,\nlist\n(\n)\nfor\ni\nin\nrange\n(\nlen\n(\nsequence\n)\n)\n:\n# find the end of this pattern\nend_ix\n=\ni\n+\nn_steps_in\nout_end_ix\n=\nend_ix\n+\nn_steps\n_\nout\n# check if we are beyond the sequence\nif\nout_end_ix\n>\nlen\n(\nsequence\n)\n:\nbreak\n# gather input and output parts of the pattern\nseq_x\n,\nseq_y\n=\nsequence\n[\ni\n:\nend_ix\n]\n,\nsequence\n[\nend_ix\n:\nout_end_ix\n]\nX\n.\nappend\n(\nseq_x\n)\ny\n.\nappend\n(\nseq_y\n)\nreturn\narray\n(\nX\n)\n,\narray\n(\ny\n)\n# define input sequence\nraw_seq\n=\n[\n10\n,\n20\n,\n30\n,\n40\n,\n50\n,\n60\n,\n70\n,\n80\n,\n90\n]\n# choose a number of time steps\nn_steps_in\n,\nn_steps_out\n=\n3\n,\n2\n# split into samples\nX\n,\ny\n=\nsplit_sequence\n(\nraw_seq\n,\nn_steps_in\n,\nn_steps_out\n)\n# reshape from [samples, timesteps] into [samples, timesteps, features]\nn_features\n=\n1\nX\n=\nX\n.\nreshape\n(\n(\nX\n.\nshape\n[\n0\n]\n,\nX\n.\nshape\n[\n1\n]\n,\nn_features\n)\n)\n# define model\nmodel\n=\nSequential\n(\n)\nmodel\n.\nadd\n(\nLSTM\n(\n100\n,\nactivation\n=\n'relu'\n,\nreturn_sequences\n=\nTrue\n,\ninput_shape\n=\n(\nn_steps_in\n,\nn_features\n)\n)\n)\nmodel\n.\nadd\n(\nLSTM\n(\n100\n,\nactivation\n=\n'relu'\n)\n)\nmodel\n.\nadd\n(\nDense\n(\nn_steps_out\n)\n)\nmodel\n.\ncompile\n(\noptimizer\n=\n'adam'\n,\nloss\n=\n'mse'\n)\n# fit model\nmodel\n.\nfit\n(\nX\n,\ny\n,\nepochs\n=\n50\n,\nverbose\n=\n0\n)\n# demonstrate prediction\nx_input\n=\narray\n(\n[\n70\n,\n80\n,\n90\n]\n)\nx_input\n=\nx_input\n.\nreshape\n(\n(\n1\n,\nn_steps_in\n,\nn_features\n)\n)\nyhat\n=\nmodel\n.\npredict\n(\nx_input\n,\nverbose\n=\n0\n)\nprint\n(\nyhat\n)\nNote\n: Your\nresults may vary\ngiven the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.\nRunning the example forecasts and prints the next two time steps in the sequence.\n[[100.98096 113.28924]]\n1\n[[100.98096 113.28924]]\nEncoder-Decoder Model\nA model specifically developed for forecasting variable length output sequences is called the\nEncoder-Decoder LSTM\n.\nThe model was designed for prediction problems where there are both input and output sequences, so-called sequence-to-sequence, or seq2seq problems, such as translating text from one language to another.\nThis model can be used for multi-step time series forecasting.\nAs its name suggests, the model is comprised of two sub-models: the encoder and the decoder.\nThe encoder is a model responsible for reading and interpreting the input sequence. The output of the encoder is a fixed length vector that represents the model’s interpretation of the sequence. The encoder is traditionally a Vanilla LSTM model, although other encoder models can be used such as Stacked, Bidirectional, and CNN models.\n...\nmodel.add(LSTM(100, activation='relu', input_shape=(n_steps_in, n_features)))\n1\n2\n.\n.\n.\nmodel\n.\nadd\n(\nLSTM\n(\n100\n,\nactivation\n=\n'relu'\n,\ninput_shape\n=\n(\nn_steps_in\n,\nn_features\n)\n)\n)\nThe decoder uses the output of the encoder as an input.\nFirst, the fixed-length output of the encoder is repeated, once for each required time step in the output sequence.\n...\nmodel.add(RepeatVector(n_steps_out))\n1\n2\n.\n.\n.\nmodel\n.\nadd\n(\nRepeatVector\n(\nn_steps_out\n)\n)\nThis sequence is then provided to an LSTM decoder model. The model must output a value for each value in the output time step, which can be interpreted by a single output model.\n...\nmodel.add(LSTM(100, activation='relu', return_sequences=True))\n1\n2\n.\n.\n.\nmodel\n.\nadd\n(\nLSTM\n(\n100\n,\nactivation\n=\n'relu'\n,\nreturn_sequences\n=\nTrue\n)\n)\nWe can use the same output layer or layers to make each one-step prediction in the output sequence. This can be achieved by wrapping the output part of the model in a\nTimeDistributed wrapper\n.\n....\nmodel.add(TimeDistributed(Dense(1)))\n1\n2\n.\n.\n.\n.\nmodel\n.\nadd\n(\nTimeDistributed\n(\nDense\n(\n1\n)\n)\n)\nThe full definition for an Encoder-Decoder model for multi-step time series forecasting is listed below.\n...\n# define model\nmodel = Sequential()\nmodel.add(LSTM(100, activation='relu', input_shape=(n_steps_in, n_features)))\nmodel.add(RepeatVector(n_steps_out))\nmodel.add(LSTM(100, activation='relu', return_sequences=True))\nmodel.add(TimeDistributed(Dense(1)))\nmodel.compile(optimizer='adam', loss='mse')\n1\n2\n3\n4\n5\n6\n7\n8\n.\n.\n.\n# define model\nmodel\n=\nSequential\n(\n)\nmodel\n.\nadd\n(\nLSTM\n(\n100\n,\nactivation\n=\n'relu'\n,\ninput_shape\n=\n(\nn_steps_in\n,\nn_features\n)\n)\n)\nmodel\n.\nadd\n(\nRepeatVector\n(\nn_steps_out\n)\n)\nmodel\n.\nadd\n(\nLSTM\n(\n100\n,\nactivation\n=\n'relu'\n,\nreturn_sequences\n=\nTrue\n)\n)\nmodel\n.\nadd\n(\nTimeDistributed\n(\nDense\n(\n1\n)\n)\n)\nmodel\n.\ncompile\n(\noptimizer\n=\n'adam'\n,\nloss\n=\n'mse'\n)\nAs with other LSTM models, the input data must be reshaped into the expected three-dimensional shape of [\nsamples, timesteps, features\n].\n...\nX = X.reshape((X.shape[0], X.shape[1], n_features))\n1\n2\n.\n.\n.\nX\n=\nX\n.\nreshape\n(\n(\nX\n.\nshape\n[\n0\n]\n,\nX\n.\nshape\n[\n1\n]\n,\nn_features\n)\n)\nIn the case of the Encoder-Decoder model, the output, or y part, of the training dataset must also have this shape. This is because the model will predict a given number of time steps with a given number of features for each input sample.\n...\ny = y.reshape((y.shape[0], y.shape[1], n_features))\n1\n2\n.\n.\n.\ny\n=\ny\n.\nreshape\n(\n(\ny\n.\nshape\n[\n0\n]\n,\ny\n.\nshape\n[\n1\n]\n,\nn_features\n)\n)\nThe complete example of an Encoder-Decoder LSTM for multi-step time series forecasting is listed below.\n# univariate multi-step encoder-decoder lstm example\nfrom numpy import array\nfrom keras.models import Sequential\nfrom keras.layers import LSTM\nfrom keras.layers import Dense\nfrom keras.layers import RepeatVector\nfrom keras.layers import TimeDistributed\n\n# split a univariate sequence into samples\ndef split_sequence(sequence, n_steps_in, n_steps_out):\n\tX, y = list(), list()\n\tfor i in range(len(sequence)):\n\t\t# find the end of this pattern\n\t\tend_ix = i + n_steps_in\n\t\tout_end_ix = end_ix + n_steps_out\n\t\t# check if we are beyond the sequence\n\t\tif out_end_ix > len(sequence):\n\t\t\tbreak\n\t\t# gather input and output parts of the pattern\n\t\tseq_x, seq_y = sequence[i:end_ix], sequence[end_ix:out_end_ix]\n\t\tX.append(seq_x)\n\t\ty.append(seq_y)\n\treturn array(X), array(y)\n\n# define input sequence\nraw_seq = [10, 20, 30, 40, 50, 60, 70, 80, 90]\n# choose a number of time steps\nn_steps_in, n_steps_out = 3, 2\n# split into samples\nX, y = split_sequence(raw_seq, n_steps_in, n_steps_out)\n# reshape from [samples, timesteps] into [samples, timesteps, features]\nn_features = 1\nX = X.reshape((X.shape[0], X.shape[1], n_features))\ny = y.reshape((y.shape[0], y.shape[1], n_features))\n# define model\nmodel = Sequential()\nmodel.add(LSTM(100, activation='relu', input_shape=(n_steps_in, n_features)))\nmodel.add(RepeatVector(n_steps_out))\nmodel.add(LSTM(100, activation='relu', return_sequences=True))\nmodel.add(TimeDistributed(Dense(1)))\nmodel.compile(optimizer='adam', loss='mse')\n# fit model\nmodel.fit(X, y, epochs=100, verbose=0)\n# demonstrate prediction\nx_input = array([70, 80, 90])\nx_input = x_input.reshape((1, n_steps_in, n_features))\nyhat = model.predict(x_input, verbose=0)\nprint(yhat)\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n14\n15\n16\n17\n18\n19\n20\n21\n22\n23\n24\n25\n26\n27\n28\n29\n30\n31\n32\n33\n34\n35\n36\n37\n38\n39\n40\n41\n42\n43\n44\n45\n46\n47\n48\n# univariate multi-step encoder-decoder lstm example\nfrom\nnumpy\nimport\narray\nfrom\nkeras\n.\nmodels\nimport\nSequential\nfrom\nkeras\n.\nlayers\nimport\nLSTM\nfrom\nkeras\n.\nlayers\nimport\nDense\nfrom\nkeras\n.\nlayers\nimport\nRepeatVector\nfrom\nkeras\n.\nlayers\nimport\nTimeDistributed\n# split a univariate sequence into samples\ndef\nsplit_sequence\n(\nsequence\n,\nn_steps_in\n,\nn_steps_out\n)\n:\nX\n,\ny\n=\nlist\n(\n)\n,\nlist\n(\n)\nfor\ni\nin\nrange\n(\nlen\n(\nsequence\n)\n)\n:\n# find the end of this pattern\nend_ix\n=\ni\n+\nn_steps_in\nout_end_ix\n=\nend_ix\n+\nn_steps\n_\nout\n# check if we are beyond the sequence\nif\nout_end_ix\n>\nlen\n(\nsequence\n)\n:\nbreak\n# gather input and output parts of the pattern\nseq_x\n,\nseq_y\n=\nsequence\n[\ni\n:\nend_ix\n]\n,\nsequence\n[\nend_ix\n:\nout_end_ix\n]\nX\n.\nappend\n(\nseq_x\n)\ny\n.\nappend\n(\nseq_y\n)\nreturn\narray\n(\nX\n)\n,\narray\n(\ny\n)\n# define input sequence\nraw_seq\n=\n[\n10\n,\n20\n,\n30\n,\n40\n,\n50\n,\n60\n,\n70\n,\n80\n,\n90\n]\n# choose a number of time steps\nn_steps_in\n,\nn_steps_out\n=\n3\n,\n2\n# split into samples\nX\n,\ny\n=\nsplit_sequence\n(\nraw_seq\n,\nn_steps_in\n,\nn_steps_out\n)\n# reshape from [samples, timesteps] into [samples, timesteps, features]\nn_features\n=\n1\nX\n=\nX\n.\nreshape\n(\n(\nX\n.\nshape\n[\n0\n]\n,\nX\n.\nshape\n[\n1\n]\n,\nn_features\n)\n)\ny\n=\ny\n.\nreshape\n(\n(\ny\n.\nshape\n[\n0\n]\n,\ny\n.\nshape\n[\n1\n]\n,\nn_features\n)\n)\n# define model\nmodel\n=\nSequential\n(\n)\nmodel\n.\nadd\n(\nLSTM\n(\n100\n,\nactivation\n=\n'relu'\n,\ninput_shape\n=\n(\nn_steps_in\n,\nn_features\n)\n)\n)\nmodel\n.\nadd\n(\nRepeatVector\n(\nn_steps_out\n)\n)\nmodel\n.\nadd\n(\nLSTM\n(\n100\n,\nactivation\n=\n'relu'\n,\nreturn_sequences\n=\nTrue\n)\n)\nmodel\n.\nadd\n(\nTimeDistributed\n(\nDense\n(\n1\n)\n)\n)\nmodel\n.\ncompile\n(\noptimizer\n=\n'adam'\n,\nloss\n=\n'mse'\n)\n# fit model\nmodel\n.\nfit\n(\nX\n,\ny\n,\nepochs\n=\n100\n,\nverbose\n=\n0\n)\n# demonstrate prediction\nx_input\n=\narray\n(\n[\n70\n,\n80\n,\n90\n]\n)\nx_input\n=\nx_input\n.\nreshape\n(\n(\n1\n,\nn_steps_in\n,\nn_features\n)\n)\nyhat\n=\nmodel\n.\npredict\n(\nx_input\n,\nverbose\n=\n0\n)\nprint\n(\nyhat\n)\nNote\n: Your\nresults may vary\ngiven the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.\nRunning the example forecasts and prints the next two time steps in the sequence.\n[[[101.9736  \n  [116.213615]]]\n1\n2\n[[[101.9736\n[116.213615]]]\nMultivariate Multi-Step LSTM Models\nIn the previous sections, we have looked at univariate, multivariate, and multi-step time series forecasting.\nIt is possible to mix and match the different types of LSTM models presented so far for the different problems. This too applies to time series forecasting problems that involve multivariate and multi-step forecasting, but it may be a little more challenging.\nIn this section, we will provide short examples of data preparation and modeling for multivariate multi-step time series forecasting as a template to ease this challenge, specifically:\nMultiple Input Multi-Step Output.\nMultiple Parallel Input and Multi-Step Output.\nPerhaps the biggest stumbling block is in the preparation of data, so this is where we will focus our attention.\nMultiple Input Multi-Step Output\nThere are those multivariate time series forecasting problems where the output series is separate but dependent upon the input time series, and multiple time steps are required for the output series.\nFor example, consider our multivariate time series from a prior section:\n[[ 10  15  25]\n [ 20  25  45]\n [ 30  35  65]\n [ 40  45  85]\n [ 50  55 105]\n [ 60  65 125]\n [ 70  75 145]\n [ 80  85 165]\n [ 90  95 185]]\n1\n2\n3\n4\n5\n6\n7\n8\n9\n[[ 10  15  25]\n[ 20  25  45]\n[ 30  35  65]\n[ 40  45  85]\n[ 50  55 105]\n[ 60  65 125]\n[ 70  75 145]\n[ 80  85 165]\n[ 90  95 185]]\nWe may use three prior time steps of each of the two input time series to predict two time steps of the output time series.\nInput:\n10, 15\n20, 25\n30, 35\n1\n2\n3\n10, 15\n20, 25\n30, 35\nOutput:\n65\n85\n1\n2\n65\n85\nThe\nsplit_sequences()\nfunction below implements this behavior.\n# split a multivariate sequence into samples\ndef split_sequences(sequences, n_steps_in, n_steps_out):\n\tX, y = list(), list()\n\tfor i in range(len(sequences)):\n\t\t# find the end of this pattern\n\t\tend_ix = i + n_steps_in\n\t\tout_end_ix = end_ix + n_steps_out-1\n\t\t# check if we are beyond the dataset\n\t\tif out_end_ix > len(sequences):\n\t\t\tbreak\n\t\t# gather input and output parts of the pattern\n\t\tseq_x, seq_y = sequences[i:end_ix, :-1], sequences[end_ix-1:out_end_ix, -1]\n\t\tX.append(seq_x)\n\t\ty.append(seq_y)\n\treturn array(X), array(y)\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n14\n15\n# split a multivariate sequence into samples\ndef\nsplit_sequences\n(\nsequences\n,\nn_steps_in\n,\nn_steps_out\n)\n:\nX\n,\ny\n=\nlist\n(\n)\n,\nlist\n(\n)\nfor\ni\nin\nrange\n(\nlen\n(\nsequences\n)\n)\n:\n# find the end of this pattern\nend_ix\n=\ni\n+\nn_steps_in\nout_end_ix\n=\nend_ix\n+\nn_steps_out\n-\n1\n# check if we are beyond the dataset\nif\nout_end_ix\n>\nlen\n(\nsequences\n)\n:\nbreak\n# gather input and output parts of the pattern\nseq_x\n,\nseq_y\n=\nsequences\n[\ni\n:\nend_ix\n,\n:\n-\n1\n]\n,\nsequences\n[\nend_ix\n-\n1\n:\nout_end_ix\n,\n-\n1\n]\nX\n.\nappend\n(\nseq_x\n)\ny\n.\nappend\n(\nseq_y\n)\nreturn\narray\n(\nX\n)\n,\narray\n(\ny\n)\nWe can demonstrate this on our contrived dataset.\nThe complete example is listed below.\n# multivariate multi-step data preparation\nfrom numpy import array\nfrom numpy import hstack\n\n# split a multivariate sequence into samples\ndef split_sequences(sequences, n_steps_in, n_steps_out):\n\tX, y = list(), list()\n\tfor i in range(len(sequences)):\n\t\t# find the end of this pattern\n\t\tend_ix = i + n_steps_in\n\t\tout_end_ix = end_ix + n_steps_out-1\n\t\t# check if we are beyond the dataset\n\t\tif out_end_ix > len(sequences):\n\t\t\tbreak\n\t\t# gather input and output parts of the pattern\n\t\tseq_x, seq_y = sequences[i:end_ix, :-1], sequences[end_ix-1:out_end_ix, -1]\n\t\tX.append(seq_x)\n\t\ty.append(seq_y)\n\treturn array(X), array(y)\n\n# define input sequence\nin_seq1 = array([10, 20, 30, 40, 50, 60, 70, 80, 90])\nin_seq2 = array([15, 25, 35, 45, 55, 65, 75, 85, 95])\nout_seq = array([in_seq1[i]+in_seq2[i] for i in range(len(in_seq1))])\n# convert to [rows, columns] structure\nin_seq1 = in_seq1.reshape((len(in_seq1), 1))\nin_seq2 = in_seq2.reshape((len(in_seq2), 1))\nout_seq = out_seq.reshape((len(out_seq), 1))\n# horizontally stack columns\ndataset = hstack((in_seq1, in_seq2, out_seq))\n# choose a number of time steps\nn_steps_in, n_steps_out = 3, 2\n# covert into input/output\nX, y = split_sequences(dataset, n_steps_in, n_steps_out)\nprint(X.shape, y.shape)\n# summarize the data\nfor i in range(len(X)):\n\tprint(X[i], y[i])\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n14\n15\n16\n17\n18\n19\n20\n21\n22\n23\n24\n25\n26\n27\n28\n29\n30\n31\n32\n33\n34\n35\n36\n37\n38\n# multivariate multi-step data preparation\nfrom\nnumpy\nimport\narray\nfrom\nnumpy\nimport\nhstack\n# split a multivariate sequence into samples\ndef\nsplit_sequences\n(\nsequences\n,\nn_steps_in\n,\nn_steps_out\n)\n:\nX\n,\ny\n=\nlist\n(\n)\n,\nlist\n(\n)\nfor\ni\nin\nrange\n(\nlen\n(\nsequences\n)\n)\n:\n# find the end of this pattern\nend_ix\n=\ni\n+\nn_steps_in\nout_end_ix\n=\nend_ix\n+\nn_steps_out\n-\n1\n# check if we are beyond the dataset\nif\nout_end_ix\n>\nlen\n(\nsequences\n)\n:\nbreak\n# gather input and output parts of the pattern\nseq_x\n,\nseq_y\n=\nsequences\n[\ni\n:\nend_ix\n,\n:\n-\n1\n]\n,\nsequences\n[\nend_ix\n-\n1\n:\nout_end_ix\n,\n-\n1\n]\nX\n.\nappend\n(\nseq_x\n)\ny\n.\nappend\n(\nseq_y\n)\nreturn\narray\n(\nX\n)\n,\narray\n(\ny\n)\n# define input sequence\nin_seq1\n=\narray\n(\n[\n10\n,\n20\n,\n30\n,\n40\n,\n50\n,\n60\n,\n70\n,\n80\n,\n90\n]\n)\nin_seq2\n=\narray\n(\n[\n15\n,\n25\n,\n35\n,\n45\n,\n55\n,\n65\n,\n75\n,\n85\n,\n95\n]\n)\nout_seq\n=\narray\n(\n[\nin_seq1\n[\ni\n]\n+\nin_seq2\n[\ni\n]\nfor\ni\nin\nrange\n(\nlen\n(\nin_seq1\n)\n)\n]\n)\n# convert to [rows, columns] structure\nin_seq1\n=\nin_seq1\n.\nreshape\n(\n(\nlen\n(\nin_seq1\n)\n,\n1\n)\n)\nin_seq2\n=\nin_seq2\n.\nreshape\n(\n(\nlen\n(\nin_seq2\n)\n,\n1\n)\n)\nout_seq\n=\nout_seq\n.\nreshape\n(\n(\nlen\n(\nout_seq\n)\n,\n1\n)\n)\n# horizontally stack columns\ndataset\n=\nhstack\n(\n(\nin_seq1\n,\nin_seq2\n,\nout_seq\n)\n)\n# choose a number of time steps\nn_steps_in\n,\nn_steps_out\n=\n3\n,\n2\n# covert into input/output\nX\n,\ny\n=\nsplit_sequences\n(\ndataset\n,\nn_steps_in\n,\nn_steps_out\n)\nprint\n(\nX\n.\nshape\n,\ny\n.\nshape\n)\n# summarize the data\nfor\ni\nin\nrange\n(\nlen\n(\nX\n)\n)\n:\nprint\n(\nX\n[\ni\n]\n,\ny\n[\ni\n]\n)\nRunning the example first prints the shape of the prepared training data.\nWe can see that the shape of the input portion of the samples is three-dimensional, comprised of six samples, with three time steps, and two variables for the 2 input time series.\nThe output portion of the samples is two-dimensional for the six samples and the two time steps for each sample to be predicted.\nThe prepared samples are then printed to confirm that the data was prepared as we specified.\n(6, 3, 2) (6, 2)\n\n[[10 15]\n [20 25]\n [30 35]] [65 85]\n[[20 25]\n [30 35]\n [40 45]] [ 85 105]\n[[30 35]\n [40 45]\n [50 55]] [105 125]\n[[40 45]\n [50 55]\n [60 65]] [125 145]\n[[50 55]\n [60 65]\n [70 75]] [145 165]\n[[60 65]\n [70 75]\n [80 85]] [165 185]\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n14\n15\n16\n17\n18\n19\n20\n(6, 3, 2) (6, 2)\n[[10 15]\n[20 25]\n[30 35]] [65 85]\n[[20 25]\n[30 35]\n[40 45]] [ 85 105]\n[[30 35]\n[40 45]\n[50 55]] [105 125]\n[[40 45]\n[50 55]\n[60 65]] [125 145]\n[[50 55]\n[60 65]\n[70 75]] [145 165]\n[[60 65]\n[70 75]\n[80 85]] [165 185]\nWe can now develop an LSTM model for multi-step predictions.\nA vector output or an encoder-decoder model could be used. In this case, we will demonstrate a vector output with a Stacked LSTM.\nThe complete example is listed below.\n# multivariate multi-step stacked lstm example\nfrom numpy import array\nfrom numpy import hstack\nfrom keras.models import Sequential\nfrom keras.layers import LSTM\nfrom keras.layers import Dense\n\n# split a multivariate sequence into samples\ndef split_sequences(sequences, n_steps_in, n_steps_out):\n\tX, y = list(), list()\n\tfor i in range(len(sequences)):\n\t\t# find the end of this pattern\n\t\tend_ix = i + n_steps_in\n\t\tout_end_ix = end_ix + n_steps_out-1\n\t\t# check if we are beyond the dataset\n\t\tif out_end_ix > len(sequences):\n\t\t\tbreak\n\t\t# gather input and output parts of the pattern\n\t\tseq_x, seq_y = sequences[i:end_ix, :-1], sequences[end_ix-1:out_end_ix, -1]\n\t\tX.append(seq_x)\n\t\ty.append(seq_y)\n\treturn array(X), array(y)\n\n# define input sequence\nin_seq1 = array([10, 20, 30, 40, 50, 60, 70, 80, 90])\nin_seq2 = array([15, 25, 35, 45, 55, 65, 75, 85, 95])\nout_seq = array([in_seq1[i]+in_seq2[i] for i in range(len(in_seq1))])\n# convert to [rows, columns] structure\nin_seq1 = in_seq1.reshape((len(in_seq1), 1))\nin_seq2 = in_seq2.reshape((len(in_seq2), 1))\nout_seq = out_seq.reshape((len(out_seq), 1))\n# horizontally stack columns\ndataset = hstack((in_seq1, in_seq2, out_seq))\n# choose a number of time steps\nn_steps_in, n_steps_out = 3, 2\n# covert into input/output\nX, y = split_sequences(dataset, n_steps_in, n_steps_out)\n# the dataset knows the number of features, e.g. 2\nn_features = X.shape[2]\n# define model\nmodel = Sequential()\nmodel.add(LSTM(100, activation='relu', return_sequences=True, input_shape=(n_steps_in, n_features)))\nmodel.add(LSTM(100, activation='relu'))\nmodel.add(Dense(n_steps_out))\nmodel.compile(optimizer='adam', loss='mse')\n# fit model\nmodel.fit(X, y, epochs=200, verbose=0)\n# demonstrate prediction\nx_input = array([[70, 75], [80, 85], [90, 95]])\nx_input = x_input.reshape((1, n_steps_in, n_features))\nyhat = model.predict(x_input, verbose=0)\nprint(yhat)\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n14\n15\n16\n17\n18\n19\n20\n21\n22\n23\n24\n25\n26\n27\n28\n29\n30\n31\n32\n33\n34\n35\n36\n37\n38\n39\n40\n41\n42\n43\n44\n45\n46\n47\n48\n49\n50\n51\n52\n# multivariate multi-step stacked lstm example\nfrom\nnumpy\nimport\narray\nfrom\nnumpy\nimport\nhstack\nfrom\nkeras\n.\nmodels\nimport\nSequential\nfrom\nkeras\n.\nlayers\nimport\nLSTM\nfrom\nkeras\n.\nlayers\nimport\nDense\n# split a multivariate sequence into samples\ndef\nsplit_sequences\n(\nsequences\n,\nn_steps_in\n,\nn_steps_out\n)\n:\nX\n,\ny\n=\nlist\n(\n)\n,\nlist\n(\n)\nfor\ni\nin\nrange\n(\nlen\n(\nsequences\n)\n)\n:\n# find the end of this pattern\nend_ix\n=\ni\n+\nn_steps_in\nout_end_ix\n=\nend_ix\n+\nn_steps_out\n-\n1\n# check if we are beyond the dataset\nif\nout_end_ix\n>\nlen\n(\nsequences\n)\n:\nbreak\n# gather input and output parts of the pattern\nseq_x\n,\nseq_y\n=\nsequences\n[\ni\n:\nend_ix\n,\n:\n-\n1\n]\n,\nsequences\n[\nend_ix\n-\n1\n:\nout_end_ix\n,\n-\n1\n]\nX\n.\nappend\n(\nseq_x\n)\ny\n.\nappend\n(\nseq_y\n)\nreturn\narray\n(\nX\n)\n,\narray\n(\ny\n)\n# define input sequence\nin_seq1\n=\narray\n(\n[\n10\n,\n20\n,\n30\n,\n40\n,\n50\n,\n60\n,\n70\n,\n80\n,\n90\n]\n)\nin_seq2\n=\narray\n(\n[\n15\n,\n25\n,\n35\n,\n45\n,\n55\n,\n65\n,\n75\n,\n85\n,\n95\n]\n)\nout_seq\n=\narray\n(\n[\nin_seq1\n[\ni\n]\n+\nin_seq2\n[\ni\n]\nfor\ni\nin\nrange\n(\nlen\n(\nin_seq1\n)\n)\n]\n)\n# convert to [rows, columns] structure\nin_seq1\n=\nin_seq1\n.\nreshape\n(\n(\nlen\n(\nin_seq1\n)\n,\n1\n)\n)\nin_seq2\n=\nin_seq2\n.\nreshape\n(\n(\nlen\n(\nin_seq2\n)\n,\n1\n)\n)\nout_seq\n=\nout_seq\n.\nreshape\n(\n(\nlen\n(\nout_seq\n)\n,\n1\n)\n)\n# horizontally stack columns\ndataset\n=\nhstack\n(\n(\nin_seq1\n,\nin_seq2\n,\nout_seq\n)\n)\n# choose a number of time steps\nn_steps_in\n,\nn_steps_out\n=\n3\n,\n2\n# covert into input/output\nX\n,\ny\n=\nsplit_sequences\n(\ndataset\n,\nn_steps_in\n,\nn_steps_out\n)\n# the dataset knows the number of features, e.g. 2\nn_features\n=\nX\n.\nshape\n[\n2\n]\n# define model\nmodel\n=\nSequential\n(\n)\nmodel\n.\nadd\n(\nLSTM\n(\n100\n,\nactivation\n=\n'relu'\n,\nreturn_sequences\n=\nTrue\n,\ninput_shape\n=\n(\nn_steps_in\n,\nn_features\n)\n)\n)\nmodel\n.\nadd\n(\nLSTM\n(\n100\n,\nactivation\n=\n'relu'\n)\n)\nmodel\n.\nadd\n(\nDense\n(\nn_steps_out\n)\n)\nmodel\n.\ncompile\n(\noptimizer\n=\n'adam'\n,\nloss\n=\n'mse'\n)\n# fit model\nmodel\n.\nfit\n(\nX\n,\ny\n,\nepochs\n=\n200\n,\nverbose\n=\n0\n)\n# demonstrate prediction\nx_input\n=\narray\n(\n[\n[\n70\n,\n75\n]\n,\n[\n80\n,\n85\n]\n,\n[\n90\n,\n95\n]\n]\n)\nx_input\n=\nx_input\n.\nreshape\n(\n(\n1\n,\nn_steps_in\n,\nn_features\n)\n)\nyhat\n=\nmodel\n.\npredict\n(\nx_input\n,\nverbose\n=\n0\n)\nprint\n(\nyhat\n)\nRunning the example fits the model and predicts the next two time steps of the output sequence beyond the dataset.\nWe would expect the next two steps to be: [185, 205]\nNote\n: Your\nresults may vary\ngiven the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.\nIt is a challenging framing of the problem with very little data, and the arbitrarily configured version of the model gets close.\n[[188.70619 210.16513]]\n1\n[[188.70619 210.16513]]\nMultiple Parallel Input and Multi-Step Output\nA problem with parallel time series may require the prediction of multiple time steps of each time series.\nFor example, consider our multivariate time series from a prior section:\n[[ 10  15  25]\n [ 20  25  45]\n [ 30  35  65]\n [ 40  45  85]\n [ 50  55 105]\n [ 60  65 125]\n [ 70  75 145]\n [ 80  85 165]\n [ 90  95 185]]\n1\n2\n3\n4\n5\n6\n7\n8\n9\n[[ 10  15  25]\n[ 20  25  45]\n[ 30  35  65]\n[ 40  45  85]\n[ 50  55 105]\n[ 60  65 125]\n[ 70  75 145]\n[ 80  85 165]\n[ 90  95 185]]\nWe may use the last three time steps from each of the three time series as input to the model and predict the next time steps of each of the three time series as output.\nThe first sample in the training dataset would be the following.\nInput:\n10, 15, 25\n20, 25, 45\n30, 35, 65\n1\n2\n3\n10, 15, 25\n20, 25, 45\n30, 35, 65\nOutput:\n40, 45, 85\n50, 55, 105\n1\n2\n40, 45, 85\n50, 55, 105\nThe\nsplit_sequences()\nfunction below implements this behavior.\n# split a multivariate sequence into samples\ndef split_sequences(sequences, n_steps_in, n_steps_out):\n\tX, y = list(), list()\n\tfor i in range(len(sequences)):\n\t\t# find the end of this pattern\n\t\tend_ix = i + n_steps_in\n\t\tout_end_ix = end_ix + n_steps_out\n\t\t# check if we are beyond the dataset\n\t\tif out_end_ix > len(sequences):\n\t\t\tbreak\n\t\t# gather input and output parts of the pattern\n\t\tseq_x, seq_y = sequences[i:end_ix, :], sequences[end_ix:out_end_ix, :]\n\t\tX.append(seq_x)\n\t\ty.append(seq_y)\n\treturn array(X), array(y)\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n14\n15\n# split a multivariate sequence into samples\ndef\nsplit_sequences\n(\nsequences\n,\nn_steps_in\n,\nn_steps_out\n)\n:\nX\n,\ny\n=\nlist\n(\n)\n,\nlist\n(\n)\nfor\ni\nin\nrange\n(\nlen\n(\nsequences\n)\n)\n:\n# find the end of this pattern\nend_ix\n=\ni\n+\nn_steps_in\nout_end_ix\n=\nend_ix\n+\nn_steps\n_\nout\n# check if we are beyond the dataset\nif\nout_end_ix\n>\nlen\n(\nsequences\n)\n:\nbreak\n# gather input and output parts of the pattern\nseq_x\n,\nseq_y\n=\nsequences\n[\ni\n:\nend_ix\n,\n:\n]\n,\nsequences\n[\nend_ix\n:\nout_end_ix\n,\n:\n]\nX\n.\nappend\n(\nseq_x\n)\ny\n.\nappend\n(\nseq_y\n)\nreturn\narray\n(\nX\n)\n,\narray\n(\ny\n)\nWe can demonstrate this function on the small contrived dataset.\nThe complete example is listed below.\n# multivariate multi-step data preparation\nfrom numpy import array\nfrom numpy import hstack\nfrom keras.models import Sequential\nfrom keras.layers import LSTM\nfrom keras.layers import Dense\nfrom keras.layers import RepeatVector\nfrom keras.layers import TimeDistributed\n\n# split a multivariate sequence into samples\ndef split_sequences(sequences, n_steps_in, n_steps_out):\n\tX, y = list(), list()\n\tfor i in range(len(sequences)):\n\t\t# find the end of this pattern\n\t\tend_ix = i + n_steps_in\n\t\tout_end_ix = end_ix + n_steps_out\n\t\t# check if we are beyond the dataset\n\t\tif out_end_ix > len(sequences):\n\t\t\tbreak\n\t\t# gather input and output parts of the pattern\n\t\tseq_x, seq_y = sequences[i:end_ix, :], sequences[end_ix:out_end_ix, :]\n\t\tX.append(seq_x)\n\t\ty.append(seq_y)\n\treturn array(X), array(y)\n\n# define input sequence\nin_seq1 = array([10, 20, 30, 40, 50, 60, 70, 80, 90])\nin_seq2 = array([15, 25, 35, 45, 55, 65, 75, 85, 95])\nout_seq = array([in_seq1[i]+in_seq2[i] for i in range(len(in_seq1))])\n# convert to [rows, columns] structure\nin_seq1 = in_seq1.reshape((len(in_seq1), 1))\nin_seq2 = in_seq2.reshape((len(in_seq2), 1))\nout_seq = out_seq.reshape((len(out_seq), 1))\n# horizontally stack columns\ndataset = hstack((in_seq1, in_seq2, out_seq))\n# choose a number of time steps\nn_steps_in, n_steps_out = 3, 2\n# covert into input/output\nX, y = split_sequences(dataset, n_steps_in, n_steps_out)\nprint(X.shape, y.shape)\n# summarize the data\nfor i in range(len(X)):\n\tprint(X[i], y[i])\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n14\n15\n16\n17\n18\n19\n20\n21\n22\n23\n24\n25\n26\n27\n28\n29\n30\n31\n32\n33\n34\n35\n36\n37\n38\n39\n40\n41\n42\n43\n# multivariate multi-step data preparation\nfrom\nnumpy\nimport\narray\nfrom\nnumpy\nimport\nhstack\nfrom\nkeras\n.\nmodels\nimport\nSequential\nfrom\nkeras\n.\nlayers\nimport\nLSTM\nfrom\nkeras\n.\nlayers\nimport\nDense\nfrom\nkeras\n.\nlayers\nimport\nRepeatVector\nfrom\nkeras\n.\nlayers\nimport\nTimeDistributed\n# split a multivariate sequence into samples\ndef\nsplit_sequences\n(\nsequences\n,\nn_steps_in\n,\nn_steps_out\n)\n:\nX\n,\ny\n=\nlist\n(\n)\n,\nlist\n(\n)\nfor\ni\nin\nrange\n(\nlen\n(\nsequences\n)\n)\n:\n# find the end of this pattern\nend_ix\n=\ni\n+\nn_steps_in\nout_end_ix\n=\nend_ix\n+\nn_steps\n_\nout\n# check if we are beyond the dataset\nif\nout_end_ix\n>\nlen\n(\nsequences\n)\n:\nbreak\n# gather input and output parts of the pattern\nseq_x\n,\nseq_y\n=\nsequences\n[\ni\n:\nend_ix\n,\n:\n]\n,\nsequences\n[\nend_ix\n:\nout_end_ix\n,\n:\n]\nX\n.\nappend\n(\nseq_x\n)\ny\n.\nappend\n(\nseq_y\n)\nreturn\narray\n(\nX\n)\n,\narray\n(\ny\n)\n# define input sequence\nin_seq1\n=\narray\n(\n[\n10\n,\n20\n,\n30\n,\n40\n,\n50\n,\n60\n,\n70\n,\n80\n,\n90\n]\n)\nin_seq2\n=\narray\n(\n[\n15\n,\n25\n,\n35\n,\n45\n,\n55\n,\n65\n,\n75\n,\n85\n,\n95\n]\n)\nout_seq\n=\narray\n(\n[\nin_seq1\n[\ni\n]\n+\nin_seq2\n[\ni\n]\nfor\ni\nin\nrange\n(\nlen\n(\nin_seq1\n)\n)\n]\n)\n# convert to [rows, columns] structure\nin_seq1\n=\nin_seq1\n.\nreshape\n(\n(\nlen\n(\nin_seq1\n)\n,\n1\n)\n)\nin_seq2\n=\nin_seq2\n.\nreshape\n(\n(\nlen\n(\nin_seq2\n)\n,\n1\n)\n)\nout_seq\n=\nout_seq\n.\nreshape\n(\n(\nlen\n(\nout_seq\n)\n,\n1\n)\n)\n# horizontally stack columns\ndataset\n=\nhstack\n(\n(\nin_seq1\n,\nin_seq2\n,\nout_seq\n)\n)\n# choose a number of time steps\nn_steps_in\n,\nn_steps_out\n=\n3\n,\n2\n# covert into input/output\nX\n,\ny\n=\nsplit_sequences\n(\ndataset\n,\nn_steps_in\n,\nn_steps_out\n)\nprint\n(\nX\n.\nshape\n,\ny\n.\nshape\n)\n# summarize the data\nfor\ni\nin\nrange\n(\nlen\n(\nX\n)\n)\n:\nprint\n(\nX\n[\ni\n]\n,\ny\n[\ni\n]\n)\nRunning the example first prints the shape of the prepared training dataset.\nWe can see that both the input (X) and output (Y) elements of the dataset are three dimensional for the number of samples, time steps, and variables or parallel time series respectively.\nThe input and output elements of each series are then printed side by side so that we can confirm that the data was prepared as we expected.\n(5, 3, 3) (5, 2, 3)\n\n[[10 15 25]\n [20 25 45]\n [30 35 65]] [[ 40  45  85]\n [ 50  55 105]]\n[[20 25 45]\n [30 35 65]\n [40 45 85]] [[ 50  55 105]\n [ 60  65 125]]\n[[ 30  35  65]\n [ 40  45  85]\n [ 50  55 105]] [[ 60  65 125]\n [ 70  75 145]]\n[[ 40  45  85]\n [ 50  55 105]\n [ 60  65 125]] [[ 70  75 145]\n [ 80  85 165]]\n[[ 50  55 105]\n [ 60  65 125]\n [ 70  75 145]] [[ 80  85 165]\n [ 90  95 185]]\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n14\n15\n16\n17\n18\n19\n20\n21\n22\n(5, 3, 3) (5, 2, 3)\n[[10 15 25]\n[20 25 45]\n[30 35 65]] [[ 40  45  85]\n[ 50  55 105]]\n[[20 25 45]\n[30 35 65]\n[40 45 85]] [[ 50  55 105]\n[ 60  65 125]]\n[[ 30  35  65]\n[ 40  45  85]\n[ 50  55 105]] [[ 60  65 125]\n[ 70  75 145]]\n[[ 40  45  85]\n[ 50  55 105]\n[ 60  65 125]] [[ 70  75 145]\n[ 80  85 165]]\n[[ 50  55 105]\n[ 60  65 125]\n[ 70  75 145]] [[ 80  85 165]\n[ 90  95 185]]\nWe can use either the Vector Output or Encoder-Decoder LSTM to model this problem. In this case, we will use the Encoder-Decoder model.\nThe complete example is listed below.\n# multivariate multi-step encoder-decoder lstm example\nfrom numpy import array\nfrom numpy import hstack\nfrom keras.models import Sequential\nfrom keras.layers import LSTM\nfrom keras.layers import Dense\nfrom keras.layers import RepeatVector\nfrom keras.layers import TimeDistributed\n\n# split a multivariate sequence into samples\ndef split_sequences(sequences, n_steps_in, n_steps_out):\n\tX, y = list(), list()\n\tfor i in range(len(sequences)):\n\t\t# find the end of this pattern\n\t\tend_ix = i + n_steps_in\n\t\tout_end_ix = end_ix + n_steps_out\n\t\t# check if we are beyond the dataset\n\t\tif out_end_ix > len(sequences):\n\t\t\tbreak\n\t\t# gather input and output parts of the pattern\n\t\tseq_x, seq_y = sequences[i:end_ix, :], sequences[end_ix:out_end_ix, :]\n\t\tX.append(seq_x)\n\t\ty.append(seq_y)\n\treturn array(X), array(y)\n\n# define input sequence\nin_seq1 = array([10, 20, 30, 40, 50, 60, 70, 80, 90])\nin_seq2 = array([15, 25, 35, 45, 55, 65, 75, 85, 95])\nout_seq = array([in_seq1[i]+in_seq2[i] for i in range(len(in_seq1))])\n# convert to [rows, columns] structure\nin_seq1 = in_seq1.reshape((len(in_seq1), 1))\nin_seq2 = in_seq2.reshape((len(in_seq2), 1))\nout_seq = out_seq.reshape((len(out_seq), 1))\n# horizontally stack columns\ndataset = hstack((in_seq1, in_seq2, out_seq))\n# choose a number of time steps\nn_steps_in, n_steps_out = 3, 2\n# covert into input/output\nX, y = split_sequences(dataset, n_steps_in, n_steps_out)\n# the dataset knows the number of features, e.g. 2\nn_features = X.shape[2]\n# define model\nmodel = Sequential()\nmodel.add(LSTM(200, activation='relu', input_shape=(n_steps_in, n_features)))\nmodel.add(RepeatVector(n_steps_out))\nmodel.add(LSTM(200, activation='relu', return_sequences=True))\nmodel.add(TimeDistributed(Dense(n_features)))\nmodel.compile(optimizer='adam', loss='mse')\n# fit model\nmodel.fit(X, y, epochs=300, verbose=0)\n# demonstrate prediction\nx_input = array([[60, 65, 125], [70, 75, 145], [80, 85, 165]])\nx_input = x_input.reshape((1, n_steps_in, n_features))\nyhat = model.predict(x_input, verbose=0)\nprint(yhat)\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n14\n15\n16\n17\n18\n19\n20\n21\n22\n23\n24\n25\n26\n27\n28\n29\n30\n31\n32\n33\n34\n35\n36\n37\n38\n39\n40\n41\n42\n43\n44\n45\n46\n47\n48\n49\n50\n51\n52\n53\n54\n55\n# multivariate multi-step encoder-decoder lstm example\nfrom\nnumpy\nimport\narray\nfrom\nnumpy\nimport\nhstack\nfrom\nkeras\n.\nmodels\nimport\nSequential\nfrom\nkeras\n.\nlayers\nimport\nLSTM\nfrom\nkeras\n.\nlayers\nimport\nDense\nfrom\nkeras\n.\nlayers\nimport\nRepeatVector\nfrom\nkeras\n.\nlayers\nimport\nTimeDistributed\n# split a multivariate sequence into samples\ndef\nsplit_sequences\n(\nsequences\n,\nn_steps_in\n,\nn_steps_out\n)\n:\nX\n,\ny\n=\nlist\n(\n)\n,\nlist\n(\n)\nfor\ni\nin\nrange\n(\nlen\n(\nsequences\n)\n)\n:\n# find the end of this pattern\nend_ix\n=\ni\n+\nn_steps_in\nout_end_ix\n=\nend_ix\n+\nn_steps\n_\nout\n# check if we are beyond the dataset\nif\nout_end_ix\n>\nlen\n(\nsequences\n)\n:\nbreak\n# gather input and output parts of the pattern\nseq_x\n,\nseq_y\n=\nsequences\n[\ni\n:\nend_ix\n,\n:\n]\n,\nsequences\n[\nend_ix\n:\nout_end_ix\n,\n:\n]\nX\n.\nappend\n(\nseq_x\n)\ny\n.\nappend\n(\nseq_y\n)\nreturn\narray\n(\nX\n)\n,\narray\n(\ny\n)\n# define input sequence\nin_seq1\n=\narray\n(\n[\n10\n,\n20\n,\n30\n,\n40\n,\n50\n,\n60\n,\n70\n,\n80\n,\n90\n]\n)\nin_seq2\n=\narray\n(\n[\n15\n,\n25\n,\n35\n,\n45\n,\n55\n,\n65\n,\n75\n,\n85\n,\n95\n]\n)\nout_seq\n=\narray\n(\n[\nin_seq1\n[\ni\n]\n+\nin_seq2\n[\ni\n]\nfor\ni\nin\nrange\n(\nlen\n(\nin_seq1\n)\n)\n]\n)\n# convert to [rows, columns] structure\nin_seq1\n=\nin_seq1\n.\nreshape\n(\n(\nlen\n(\nin_seq1\n)\n,\n1\n)\n)\nin_seq2\n=\nin_seq2\n.\nreshape\n(\n(\nlen\n(\nin_seq2\n)\n,\n1\n)\n)\nout_seq\n=\nout_seq\n.\nreshape\n(\n(\nlen\n(\nout_seq\n)\n,\n1\n)\n)\n# horizontally stack columns\ndataset\n=\nhstack\n(\n(\nin_seq1\n,\nin_seq2\n,\nout_seq\n)\n)\n# choose a number of time steps\nn_steps_in\n,\nn_steps_out\n=\n3\n,\n2\n# covert into input/output\nX\n,\ny\n=\nsplit_sequences\n(\ndataset\n,\nn_steps_in\n,\nn_steps_out\n)\n# the dataset knows the number of features, e.g. 2\nn_features\n=\nX\n.\nshape\n[\n2\n]\n# define model\nmodel\n=\nSequential\n(\n)\nmodel\n.\nadd\n(\nLSTM\n(\n200\n,\nactivation\n=\n'relu'\n,\ninput_shape\n=\n(\nn_steps_in\n,\nn_features\n)\n)\n)\nmodel\n.\nadd\n(\nRepeatVector\n(\nn_steps_out\n)\n)\nmodel\n.\nadd\n(\nLSTM\n(\n200\n,\nactivation\n=\n'relu'\n,\nreturn_sequences\n=\nTrue\n)\n)\nmodel\n.\nadd\n(\nTimeDistributed\n(\nDense\n(\nn_features\n)\n)\n)\nmodel\n.\ncompile\n(\noptimizer\n=\n'adam'\n,\nloss\n=\n'mse'\n)\n# fit model\nmodel\n.\nfit\n(\nX\n,\ny\n,\nepochs\n=\n300\n,\nverbose\n=\n0\n)\n# demonstrate prediction\nx_input\n=\narray\n(\n[\n[\n60\n,\n65\n,\n125\n]\n,\n[\n70\n,\n75\n,\n145\n]\n,\n[\n80\n,\n85\n,\n165\n]\n]\n)\nx_input\n=\nx_input\n.\nreshape\n(\n(\n1\n,\nn_steps_in\n,\nn_features\n)\n)\nyhat\n=\nmodel\n.\npredict\n(\nx_input\n,\nverbose\n=\n0\n)\nprint\n(\nyhat\n)\nRunning the example fits the model and predicts the values for each of the three time steps for the next two time steps beyond the end of the dataset.\nWe would expect the values for these series and time steps to be as follows:\n90, 95, 185\n100, 105, 205\n1\n2\n90, 95, 185\n100, 105, 205\nNote\n: Your\nresults may vary\ngiven the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.\nWe can see that the model forecast gets reasonably close to the expected values.\n[[[ 91.86044   97.77231  189.66768 ]\n  [103.299355 109.18123  212.6863  ]]]\n1\n2\n[[[ 91.86044   97.77231  189.66768 ]\n[103.299355 109.18123  212.6863  ]]]\nFurther Reading\nLong short-term memory\n, Wikipedia.\nDeep Learning for Time Series Forecasting\n(my book)\nSummary\nIn this tutorial, you discovered how to develop a suite of LSTM models for a range of standard time series forecasting problems.\nSpecifically, you learned:\nHow to develop LSTM models for univariate time series forecasting.\nHow to develop LSTM models for multivariate time series forecasting.\nHow to develop LSTM models for multi-step time series forecasting.\nDo you have any questions?\nAsk your questions in the comments below and I will do my best to answer.\nDevelop Deep Learning models for Time Series Today!\nDevelop Your Own Forecasting models in Minutes\n...with just a few lines of python code\nDiscover how in my new Ebook:\nDeep Learning for Time Series Forecasting\nIt provides\nself-study tutorials\non topics like:\nCNNs\n,\nLSTMs\n,\nMultivariate Forecasting\n,\nMulti-Step Forecasting\nand much more...\nFinally Bring Deep Learning to your Time Series Forecasting Projects\nSkip the Academics. Just Results.\nSee What's Inside\nShare\nPost\nShare\nMore On This Topic\nMulti-Step LSTM Time Series Forecasting Models for…\nMastering Time Series Forecasting: From ARIMA to LSTM\nLSTM Model Architecture for Rare Event Time Series…\nWeight Regularization with LSTM Networks for Time…\nDropout with LSTM Networks for Time Series Forecasting\nInstability of Online Learning for Stateful LSTM for…\nAbout Jason Brownlee\nJason Brownlee, PhD is a machine learning specialist who teaches developers how to get results with modern machine learning methods via hands-on tutorials.\nView all posts by Jason Brownlee\n→",
  "metadata": {
    "title": "How to Develop LSTM Models for Time Series Forecasting - MachineLearningMastery.com",
    "description": null,
    "keywords": null,
    "author": "Jason Brownlee",
    "og:locale": "en_US",
    "og:type": "article",
    "og:title": "How to Develop LSTM Models for Time Series Forecasting - MachineLearningMastery.com",
    "og:description": "Long Short-Term Memory networks, or LSTMs for short, can be applied to time series forecasting. There are many types of LSTM models that can be used for each specific type of time series forecasting problem. In this tutorial, you will discover how to develop a suite of LSTM models for a range of standard time series forecasting problems. The objective…",
    "og:url": "https://machinelearningmastery.com/how-to-develop-lstm-models-for-time-series-forecasting/",
    "og:site_name": "MachineLearningMastery.com",
    "og:image": "https://machinelearningmastery.com/wp-content/uploads/2018/11/How-to-Develop-LSTM-Models-for-Time-Series-Forecasting.jpg",
    "og:image:width": "640",
    "og:image:height": "360",
    "og:image:type": "image/jpeg",
    "twitter:label1": "Written by",
    "twitter:data1": "Jason Brownlee",
    "twitter:label2": "Est. reading time",
    "twitter:data2": "49 minutes",
    "twitter:card": "summary_large_image",
    "twitter:description": "Long Short-Term Memory networks, or LSTMs for short, can be applied to time series forecasting. There are many types of LSTM models that can be used for each specific type of time series forecasting problem. In this tutorial, you will discover how to develop a suite of LSTM models for a range of standard time series forecasting problems. The objective…",
    "twitter:title": "How to Develop LSTM Models for Time Series Forecasting - MachineLearningMastery.com",
    "article:publisher": "https://www.facebook.com/MachineLearningMastery/",
    "article:author": "https://www.facebook.com/MachineLearningMastery/",
    "article:published_time": "2018-11-13T18:00:06+00:00",
    "article:modified_time": "2020-08-27T23:09:06+00:00"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [],
  "images": [
    {
      "status": "ok",
      "url": "https://machinelearningmastery.com/wp-content/uploads/2018/11/How-to-Develop-LSTM-Models-for-Time-Series-Forecasting.jpg",
      "bytes": 57606,
      "reused": true,
      "local_path": "attachments/e9465448bb83/images/000.jpg",
      "src": "https://machinelearningmastery.com/wp-content/uploads/2018/11/How-to-Develop-LSTM-Models-for-Time-Series-Forecasting.jpg",
      "alt": "How to Develop LSTM Models for Time Series Forecasting",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://machinelearningmastery.com/wp-content/uploads/2022/11/DL4TSF-220.png",
      "bytes": 4455,
      "reused": false,
      "local_path": "attachments/e9465448bb83/images/body-1235302e1544cf453f05.png",
      "src": "https://machinelearningmastery.com/wp-content/uploads/2022/11/DL4TSF-220.png",
      "alt": "Deep Learning for Time Series Forecasting",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://machinelearningmastery.com/wp-content/uploads/2018/10/How-to-Develop-LSTM-Models-for-Multi-Step-Time-Series-Forecasting-of-Household-Power-Consumption.jpg",
      "bytes": 47777,
      "reused": true,
      "local_path": "attachments/e9465448bb83/images/003.jpg",
      "src": "https://machinelearningmastery.com/wp-content/uploads/2018/10/How-to-Develop-LSTM-Models-for-Multi-Step-Time-Series-Forecasting-of-Household-Power-Consumption.jpg",
      "alt": "How to Develop LSTM Models for Multi-Step Time Series Forecasting of Household Power Consumption",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://machinelearningmastery.com/wp-content/uploads/2025/03/mlm-black-grey-white-wave-design-200x200.png",
      "bytes": 11222,
      "reused": true,
      "local_path": "attachments/e9465448bb83/images/014.png",
      "src": "https://machinelearningmastery.com/wp-content/uploads/2025/03/mlm-black-grey-white-wave-design-200x200.png",
      "alt": "mlm-black-grey-white-wave-design",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://machinelearningmastery.com/wp-content/uploads/2018/11/Sliding-Window-Approach-to-Modeling-Time-Series.png",
      "bytes": 37503,
      "reused": false,
      "local_path": "attachments/e9465448bb83/images/body-ad33e5d8c27cbd7d3e0d.png",
      "src": "https://machinelearningmastery.com/wp-content/uploads/2018/11/Sliding-Window-Approach-to-Modeling-Time-Series.png",
      "alt": "Sliding Window Approach to Modeling Time Series",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://machinelearningmastery.com/wp-content/uploads/2017/03/How-to-Use-Weight-Regularization-with-LSTM-Networks-for-Time-Series-Forecasting.jpg",
      "bytes": 46303,
      "reused": false,
      "local_path": "attachments/e9465448bb83/images/body-9331ae98bbcbe2fd78c2.jpg",
      "src": "https://machinelearningmastery.com/wp-content/uploads/2017/03/How-to-Use-Weight-Regularization-with-LSTM-Networks-for-Time-Series-Forecasting.jpg",
      "alt": "How to Use Weight Regularization with LSTM Networks for Time Series Forecasting",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://machinelearningmastery.com/wp-content/uploads/2017/04/How-to-Use-Dropout-with-LSTM-Networks-for-Time-Series-Forecasting.jpg",
      "bytes": 49649,
      "reused": false,
      "local_path": "attachments/e9465448bb83/images/body-04424343870ae5ae55c5.jpg",
      "src": "https://machinelearningmastery.com/wp-content/uploads/2017/04/How-to-Use-Dropout-with-LSTM-Networks-for-Time-Series-Forecasting.jpg",
      "alt": "How to Use Dropout with LSTM Networks for Time Series Forecasting",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://machinelearningmastery.com/wp-content/uploads/2017/03/Instability-of-Online-Learning-for-Stateful-LSTM-for-Time-Series-Forecasting.jpg",
      "bytes": 48916,
      "reused": false,
      "local_path": "attachments/e9465448bb83/images/body-d0991add23f84f0d17da.jpg",
      "src": "https://machinelearningmastery.com/wp-content/uploads/2017/03/Instability-of-Online-Learning-for-Stateful-LSTM-for-Time-Series-Forecasting.jpg",
      "alt": "Instability of Online Learning for Stateful LSTM for Time Series Forecasting",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://machinelearningmastery.com/author/jasonb/",
      "label": "Jason Brownlee",
      "type": "a"
    },
    {
      "href": "https://machinelearningmastery.com/category/deep-learning-time-series/",
      "label": "Deep Learning for Time Series",
      "type": "a"
    },
    {
      "href": "https://machinelearningmastery.com/how-to-develop-lstm-models-for-time-series-forecasting/",
      "label": "991",
      "type": "a"
    },
    {
      "href": "https://machinelearningmastery.com/time-series-forecasting/",
      "label": "time series forecasting",
      "type": "a"
    },
    {
      "href": "https://machinelearningmastery.com/deep-learning-for-time-series-forecasting/",
      "label": "Deep Learning for Time Series Forecasting",
      "type": "a"
    },
    {
      "href": "https://www.flickr.com/photos/15216811@N06/6704346543/",
      "label": "N i c o l a",
      "type": "a"
    },
    {
      "href": "https://machinelearningmastery.com/adam-optimization-algorithm-for-deep-learning/",
      "label": "Adam version of stochastic gradient descent",
      "type": "a"
    },
    {
      "href": "https://machinelearningmastery.com/different-results-each-time-in-machine-learning/",
      "label": "results may vary",
      "type": "a"
    },
    {
      "href": "https://machinelearningmastery.com/develop-bidirectional-lstm-sequence-classification-python-keras/",
      "label": "Bidirectional LSTM",
      "type": "a"
    },
    {
      "href": "https://machinelearningmastery.com/cnn-long-short-term-memory-networks/",
      "label": "This hybrid model is called a CNN-LSTM",
      "type": "a"
    },
    {
      "href": "https://machinelearningmastery.com/timedistributed-layer-for-long-short-term-memory-networks-in-python/",
      "label": "TimeDistributed wrapper",
      "type": "a"
    },
    {
      "href": "https://machinelearningmastery.com/encoder-decoder-long-short-term-memory-networks/",
      "label": "Encoder-Decoder LSTM",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Long_short-term_memory",
      "label": "Long short-term memory",
      "type": "a"
    },
    {
      "href": "https://machinelearningmastery.com/how-to-develop-lstm-models-for-multi-step-time-series-forecasting-of-household-power-consumption/",
      "label": "Multi-Step LSTM Time Series Forecasting Models for…",
      "type": "a"
    },
    {
      "href": "https://machinelearningmastery.com/mastering-time-series-forecasting-from-arima-to-lstm/",
      "label": "Mastering Time Series Forecasting: From ARIMA to LSTM",
      "type": "a"
    },
    {
      "href": "https://machinelearningmastery.com/lstm-model-architecture-for-rare-event-time-series-forecasting/",
      "label": "LSTM Model Architecture for Rare Event Time Series…",
      "type": "a"
    },
    {
      "href": "https://machinelearningmastery.com/use-weight-regularization-lstm-networks-time-series-forecasting/",
      "label": "Weight Regularization with LSTM Networks for Time…",
      "type": "a"
    },
    {
      "href": "https://machinelearningmastery.com/use-dropout-lstm-networks-time-series-forecasting/",
      "label": "Dropout with LSTM Networks for Time Series Forecasting",
      "type": "a"
    },
    {
      "href": "https://machinelearningmastery.com/instability-online-learning-stateful-lstm-time-series-forecasting/",
      "label": "Instability of Online Learning for Stateful LSTM for…",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:54:22Z",
  "source_url": "https://machinelearningmastery.com/how-to-develop-lstm-models-for-time-series-forecasting/",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 107890,
  "expected_body_images": 8,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/e9465448bb83/pages/source.html",
  "local_html": "attachments/e9465448bb83/pages/article.html",
  "body_base_path": "attachments/e9465448bb83/pages/",
  "body_markdown_path": "attachments/e9465448bb83/pages/article.md"
}