{
  "source_file": "0930",
  "source_platform": "Hashnode",
  "author": "Juan Carlos Olamendy",
  "title_listed": "Gradient Descent in Deep Learning: A Complete Guide with PyTorch and Keras Examples",
  "keywords": "",
  "domain": "机器学习",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://juancolamendy.hashnode.dev/gradient-descent-in-deep-learning-a-complete-guide-with-pytorch-and-keras-examples",
  "publish_date": "2025-08-26",
  "corpus_stage": "",
  "notes": "",
  "id": "fef819142dd2",
  "collected_at": "2026-10-02T14:10:49",
  "title": "Gradient Descent in Deep Learning: A Complete Guide with PyTorch and Keras Examples",
  "body_markdown": "Gradient Descent in Deep Learning: A Complete Guide with PyTorch and Keras Examples\nPublished\nAugust 26, 2025\n•\n11\nmin read\n•\nView as Markdown\nJ\nJuan Carlos Olamendy\nSenior AI Engineer | Building & Telling Stories about AI/ML Systems | Software Engineer\nImagine you're blindfolded on a mountainside, trying to find the lowest valley. You can only feel the slope beneath your feet and take one step at a time.\nHow would you navigate to the bottom?\nThis exact scenario mirrors one of the most fundamental algorithms driving every neural network, recommendation system, and AI breakthrough you encounter daily.\nGradient descent powers the learning mechanism behind ChatGPT, image recognition systems, and autonomous vehicles.\nUnderstanding this algorithm isn't just academic curiosity—it's the key to unlocking how machines actually learn.\nToday, we'll demystify gradient descent through hands-on examples in both PyTorch and Keras, giving you the practical knowledge to implement and optimize this critical algorithm.\nWhat is Gradient Descent?\nGradient descent represents the optimization algorithm that enables neural networks to learn from data.\nThink of it as a systematic method for finding the minimum point of a function, much like rolling a ball down a hill until it reaches the bottom.\nIn machine learning, this \"hill\" is the loss function, and the \"bottom\" represents the optimal set of parameters that minimize prediction errors.\nThe algorithm works by calculating the slope (gradient) at your current position and taking steps in the direction of steepest descent.\nEvery time you see a neural network improve its accuracy during training, gradient descent operates behind the scenes.\nThe algorithm iteratively adjusts millions or billions of parameters, each step guided by mathematical precision.\nThis process transforms random initializations into sophisticated models that can recognize faces, translate languages, or predict market trends. Without gradient descent, deep learning as we know it simply wouldn't exist.\nThe beauty of gradient descent lies in its elegant simplicity. Despite being conceptually straightforward, it scales from optimizing a single neuron to training transformer models with hundreds of billions of parameters.\nThis scalability makes it the universal language of machine learning optimization. Whether you're working with computer vision, natural language processing, or reinforcement learning, gradient descent remains your fundamental tool.\nThe Mathematics Behind Gradient Descent\nThe mathematical foundation of gradient descent rests on calculus and the concept of derivatives.\nFor a function f(x), the derivative f'(x) tells us how the function changes as x increases.\nIn machine learning, we're interested in how our loss function L(θ) changes as we adjust our parameters θ. The gradient ∇L(θ) represents the vector of partial derivatives with respect to each parameter.\nThe core gradient descent update rule follows this formula: θ_new = θ_old - α × ∇L(θ)\nHere, α represents the learning rate, controlling how large steps we take.\nThe negative sign ensures we move opposite to the gradient direction, toward lower loss values.\nThis simple equation encapsulates the entire learning process of neural networks. Each parameter gets updated based on how much it contributes to the overall error.\nConsider a simple quadratic function L(w) = (w - 2)². The gradient is dL/dw = 2(w - 2). If w = 5, the gradient equals 6, indicating we should decrease w. If w = 1, the gradient equals -2, indicating we should increase w.\nThis mathematical relationship guides every parameter update in your neural network.\nTypes of Gradient Descent\nMachine learning practitioners employ three main variants of gradient descent, each with distinct computational and convergence characteristics.\nBatch Gradient Descent\ncomputes gradients using the entire training dataset for each update. This approach provides the most accurate gradient estimates but requires substantial computational resources for large datasets. The algorithm guarantees convergence to the global minimum for convex functions but processes data slowly.\nStochastic Gradient Descent (SGD)\nupdates parameters after processing each individual training example. This method introduces noise into the optimization process but enables much faster computations. The noise actually helps the algorithm escape local minima, though it makes convergence less smooth. SGD works particularly well for online learning scenarios where data arrives continuously.\nMini-batch Gradient Descent\nstrikes a balance by computing gradients on small subsets of the training data. Most deep learning frameworks default to this approach because it combines computational efficiency with gradient accuracy. Typical batch sizes range from 32 to 256 examples, depending on available memory and model complexity. This variant captures the benefits of both previous methods while mitigating their respective drawbacks.\nThe choice between these variants depends on your dataset size, computational constraints, and convergence requirements.\nLarge-scale applications almost always use mini-batch gradient descent due to its optimal resource utilization.\nImplementing Gradient Descent in PyTorch\nPyTorch provides explicit control over the gradient descent process, making it ideal for understanding the underlying mechanics.\nLet's start with a fundamental single-neuron example that demonstrates every component of the optimization process.\nThis hands-on approach reveals exactly how gradients flow and parameters update.\nimport\ntorch\nimport\ntorch.nn.functional\nas\nF\nfrom\ntorch.autograd\nimport\ngrad\n# Define our data and initial parameters\ny = torch.tensor([\n1.0\n])\n# Ground truth label\nx1 = torch.tensor([\n1.1\n])\n# Input feature\nw1 = torch.tensor([\n2.2\n], requires_grad=\nTrue\n)\n# Trainable weight\nb = torch.tensor([\n0.0\n], requires_grad=\nTrue\n)\n# Trainable bias\n# Forward pass: compute prediction\nz = x1 * w1 + b\n# Linear combination\na = torch.sigmoid(z)\n# Sigmoid activation\nprint(\nf\"Prediction:\n{a.item():\n.4\nf}\n\"\n)\n# Compute loss\nloss = F.binary_cross_entropy(a, y)\nprint(\nf\"Loss:\n{loss.item():\n.4\nf}\n\"\n)\n# Method 1: Manual gradient computation\ngrad_w1 = grad(loss, w1, retain_graph=\nTrue\n)[\n0\n]\ngrad_b = grad(loss, b, retain_graph=\nTrue\n)[\n0\n]\nprint(\nf\"Weight gradient:\n{grad_w1.item():\n.4\nf}\n\"\n)\nprint(\nf\"Bias gradient:\n{grad_b.item():\n.4\nf}\n\"\n)\nThe\nrequires_grad=True\nflag tells PyTorch to track operations on these tensors for gradient computation. During the forward pass, PyTorch builds a computational graph automatically.\nThe\ngrad()\nfunction computes gradients by traversing this graph backward. Setting\nretain_graph=True\nprevents PyTorch from destroying the graph after the first gradient computation.\n# Method 2: Automatic gradient computation (more common)\nloss.backward()\n# Compute all gradients automatically\nprint(\nf\"Weight gradient:\n{w1.grad.item():\n.4\nf}\n\"\n)\nprint(\nf\"Bias gradient:\n{b.grad.item():\n.4\nf}\n\"\n)\n# Manual parameter update\nlearning_rate =\n0.1\nwith\ntorch.no_grad():\n# Disable gradient tracking for updates\nw1 -= learning_rate * w1.grad\n    b -= learning_rate * b.grad\n# Clear gradients for next iteration\nw1.grad.zero_()\nb.grad.zero_()\nThe\n.backward()\nmethod represents the standard approach for gradient computation in PyTorch.\nIt automatically computes gradients for all tensors with\nrequires_grad=True\nand stores them in the\n.grad\nattribute.\nThe\ntorch.no\n_grad()\ncontext manager disables gradient tracking during parameter updates, preventing PyTorch from building unnecessary computational graphs.\nAlways remember to zero gradients after each update, as PyTorch accumulates gradients by default.\nComplete Training Loop in PyTorch\nA realistic PyTorch implementation requires organizing these components into a proper training loop.\nimport\ntorch\nimport\ntorch.nn\nas\nnn\nimport\ntorch.optim\nas\noptim\n# Initialize model, loss, and optimizer\nmodel = NeuralNetwork()\ncriterion = nn.BCELoss()\noptimizer = optim.SGD(model.parameters(), lr=\n0.1\n)\n# Training loop\nepochs =\n100\nfor\nepoch\nin\nrange(epochs):\n# Forward pass\npredictions = model(X)\n    loss = criterion(predictions, y)\n# Backward pass\noptimizer.zero_grad()\n# Clear previous gradients\nloss.backward()\n# Compute gradients\noptimizer.step()\n# Update parameters\nif\nepoch %\n20\n==\n0\n:\n        print(\nf\"Epoch\n{epoch}\n, Loss:\n{loss.item():\n.4\nf}\n\"\n)\nImplementing Gradient Descent in Keras\nKeras takes a higher-level approach, abstracting away much of the manual gradient computation while maintaining the same underlying principles.\nThe framework handles backpropagation, gradient computation, and parameter updates automatically.\nThis abstraction enables rapid prototyping and experimentation with complex architectures.\nimport\ntensorflow\nas\ntf\nfrom\ntensorflow\nimport\nkeras\n# Compile model (sets up training infrastructure)\nmodel.compile(\n    optimizer=keras.optimizers.SGD(learning_rate=\n0.1\n),\n    loss=\n'binary_crossentropy'\n,\n    metrics=[\n'accuracy'\n]\n)\n# Train model\nmodel.fit(\n    X, y,\n    epochs=\n100\n,\n    batch_size=\n32\n,\n    verbose=\n0\n# Reduce output for brevity\n)\nKeras's\ncompile()\nmethod configures the entire training infrastructure in a single call.\nThe optimizer parameter specifies which gradient descent variant to use, along with hyperparameters like learning rate.\nThe\nfit()\nmethod encapsulates the entire training loop, handling forward passes, loss computation, backpropagation, and parameter updates.\nThis high-level interface enables focus on model architecture and hyperparameter tuning rather than implementation details.\nManual Gradient Access in Keras\nFor researchers who need explicit gradient access, TensorFlow provides\nGradientTape\nfor manual gradient computation.\nThis functionality bridges the gap between Keras's high-level interface and PyTorch's explicit control.\n# Manual training loop with gradient access\noptimizer = keras.optimizers.SGD(learning_rate=\n0.1\n)\nfor\nepoch\nin\nrange(\n10\n):\nwith\ntf.GradientTape()\nas\ntape:\n# Forward pass\npredictions = model(X, training=\nTrue\n)\n        loss = keras.losses.binary_crossentropy(y_tensor, predictions)\n        loss = tf.reduce_mean(loss)\n# Average across batch\n# Compute gradients\ngradients = tape.gradient(loss, model.trainable_variables)\n# Apply gradients\noptimizer.apply_gradients(zip(gradients, model.trainable_variables))\nif\nepoch %\n2\n==\n0\n:\n            print(\nf\"Epoch\n{epoch}\n, Loss:\n{loss.numpy():\n.4\nf}\n\"\n)\nGradientTape\nrecords operations during the forward pass, similar to PyTorch's autograd system.\nThe\ntape.gradient()\nmethod computes gradients by differentiating the recorded computation graph.\nThe optimizer's\napply_gradients()\nmethod performs parameter updates using the computed gradients.\nThis approach provides the flexibility of manual gradient access while maintaining Keras's ecosystem benefits.\nCommon Challenges and Solutions\nLearning Rate Selection\nChoosing an appropriate learning rate represents one of the most critical decisions in gradient descent optimization.\nToo high a learning rate causes the algorithm to overshoot minima, leading to divergence or oscillation.\nToo low a learning rate results in painfully slow convergence and potential stagnation in local minima.\nThe optimal learning rate depends on the problem complexity, dataset size, and model architecture.\nSolution — Adaptive learning rate methods\nlike Adam, AdaGrad, and RMSprop automatically adjust learning rates during training.\nAdaGrad\nadapts learning rates based on the historical sum of squared gradients for each parameter. Parameters with large gradients receive smaller learning rates, while parameters with small gradients receive larger learning rates. This adaptation helps deal with sparse gradients and different parameter scales. However, AdaGrad's learning rate can decay too aggressively, eventually stopping learning entirely.\nRMSprop\naddresses AdaGrad's aggressive learning rate decay by using a moving average of squared gradients.\nAdam\ncombines the benefits of momentum with adaptive learning rates, making it widely applicable.\nAdamW\nadds weight decay to Adam, improving generalization on many tasks. Each optimizer has hyperparameters that require tuning for optimal performance on specific problems.\nThese optimizers maintain per-parameter learning rates based on historical gradient information.\nVanishing and Exploding Gradients\nDeep neural networks suffer from gradient-related pathologies that impede effective learning.\nVanishing gradients\noccur when gradients become exponentially small as they propagate backward through layers.\nExploding gradients\ncause gradients to grow exponentially large, leading to unstable training.\nBoth problems stem from the repeated multiplication of weights and derivatives through many layers.\nSolution — Gradient clipping\nlimits gradient magnitudes to prevent exploding gradients.\nProper weight initialization\nschemes like Xavier or He initialization help maintain gradient flow.\nBatch normalization\nnormalizes inputs to each layer, stabilizing the optimization landscape.\nResidual connections\nprovide gradient highways that bypass vanishing gradient problems.\nLocal Minima and Saddle Points\nThe non-convex nature of neural network loss functions creates multiple local minima and saddle points.\nTraditional concerns about local minima have been somewhat alleviated by research showing that most local minima in high-dimensional spaces have similar loss values.\nSaddle points\npresent a more significant challenge, as gradients can become very small without reaching an optimal solution.\nModern optimizers like Adam include\nmomentum methods\nthat help escape saddle points.\nStochastic noise\ninherent in mini-batch gradient descent actually helps escape poor local minima.\nMultiple random initializations\ncan be used to explore different regions of the parameter space.\nCyclical learning rates\nperiodically increase learning rates to help jump out of local minima.\nEnsemble methods\ncombine models trained from different initializations to improve overall performance.\nMomentum Methods\nMomentum accelerates gradient descent by accumulating past gradients, similar to a ball rolling downhill.\nThis technique helps the optimizer maintain direction when traversing flat regions or noisy gradients.\nStandard momentum\nuses a simple exponential moving average of past gradients.\nNesterov momentum\nprovides a more sophisticated approach by looking ahead before making updates.\nThe momentum parameter typically ranges from 0.9 to 0.99, with higher values providing more smoothing.\nMomentum helps overcome small local minima and reduces oscillation in narrow valleys.\nThe technique proves particularly effective when gradients consistently point in the same general direction. However, momentum can overshoot minima, requiring careful tuning of the momentum coefficient.\nBest Practices for Gradient Descent Implementation\nHyper-parameter Tuning Strategy\nSystematic hyper-parameter optimization significantly impacts model performance and training stability.\nLearning rate schedulers\noften provide better results than fixed learning rates throughout training. Monitor both training and validation loss to detect overfitting and adjust regularization accordingly.\nBatch size\naffects both convergence speed and gradient noise levels.\nSmaller batches\nprovide more gradient noise, which can help escape local minima but may slow convergence.\nLarger batches\nprovide more accurate gradients but require more memory and may converge to sharper minima. The optimal batch size often depends on the specific dataset and model architecture.\nMonitoring and Debugging\nEffective monitoring prevents common training failures and provides insights into optimization behavior. Track gradient norms to detect vanishing or exploding gradients before they cause training failure.\nLoss curves\nshould generally decrease smoothly, with occasional plateaus but no sustained increases.\nLearning rate schedules\nshould align with loss curve behavior—reduce learning rates when loss plateaus.\nEarly stopping\nprevents overfitting by monitoring validation loss and stopping when it stops improving.\nGradient norm monitoring\nhelps detect optimization problems before they become severe.\nWeight and activation visualizations\nprovide insights into what the model learns during training.\nLearning curves\nreveal whether the model needs more capacity, more data, or different hyper-parameters.\nMemory and Computational Efficiency\nModern deep learning models require careful memory management to train effectively.\nGradient accumulation\nsimulates larger batch sizes when memory constraints prevent loading full batches.\nMixed precision training\nreduces memory usage and increases training speed by using 16-bit floating point operations.\nCheckpoint saving\nenables recovery from training interruptions without losing progress.\nDataLoader optimization\nimproves training throughput.\nModel parallelism\ndistributes large models across multiple GPUs when they don't fit in single-GPU memory.\nEfficient data pipelines\nprevent I/O bottlenecks from limiting training speed.\nConclusion\nGradient descent stands as the fundamental algorithm that enables machine learning models to learn from data, driving virtually every breakthrough in artificial intelligence from simple linear regression to complex transformer models with billions of parameters.\nUnderstanding gradient descent provides the mathematical foundation necessary for developing, debugging, and improving machine learning systems.\nThe examples in PyTorch and Keras demonstrate how modern frameworks implement these principles while providing practical tools for real-world applications, revealing both the power and elegant simplicity of this optimization technique.\nThe journey from mathematical theory to practical implementation reveals important challenges like vanishing gradients, local minima, and hyperparameter sensitivity that require sophisticated solutions.\nAdvanced techniques like momentum, adaptive learning rates, and second-order methods address these challenges while opening new research directions.\nAs you apply gradient descent in your own projects, remember that successful implementation requires balancing theoretical understanding with practical constraints—monitor training carefully, choose appropriate optimizers for your specific problem, and experiment with different approaches.\nMaster these concepts, and you'll be well-equipped to contribute to the exciting future of artificial intelligence, where gradient descent continues to serve as the reliable foundation upon which innovation builds.\nPS:\nIf you like this article, share it with others ♻️\nWould help a lot ❤️\nAnd feel free to follow me for more content like this.\n#\ndeep-learning\n#\ngradient-descent\n#\nmachine-learning\n#\nartificial-intelligence\n#\nkeras\n#\npytorch",
  "metadata": {
    "title": "Gradient Descent in Deep Learning: A Complete Guide with PyTorch and Keras Examples",
    "description": "Imagine you're blindfolded on a mountainside, trying to find the lowest valley. You can only feel the slope beneath your feet and take one step at a time.\nHow would you navigate to the bottom?\nThis exact scenario mirrors one of the most fundamental a...",
    "keywords": null,
    "author": null,
    "og:title": "Gradient Descent in Deep Learning: A Complete Guide with PyTorch and Keras Examples",
    "og:description": "Imagine you're blindfolded on a mountainside, trying to find the lowest valley. You can only feel the slope beneath your feet and take one step at a time.\nHow would you navigate to the bottom?\nThis exact scenario mirrors one of the most fundamental a...",
    "og:url": "https://juancolamendy.hashnode.dev/gradient-descent-in-deep-learning-a-complete-guide-with-pytorch-and-keras-examples",
    "og:site_name": "Juan C Olamendy",
    "og:image": "https://cdn.hashnode.com/res/hashnode/image/upload/v1756225525421/e62e0a0b-45ba-4980-baf9-947f53e6039a.png",
    "og:image:width": "1200",
    "og:image:height": "630",
    "og:type": "article",
    "twitter:card": "summary_large_image",
    "twitter:creator": "https://twitter.com/juancolamendy",
    "twitter:title": "Gradient Descent in Deep Learning: A Complete Guide with PyTorch and Keras Examples",
    "twitter:description": "Imagine you're blindfolded on a mountainside, trying to find the lowest valley. You can only feel the slope beneath your feet and take one step at a time.\nHow would you navigate to the bottom?\nThis exact scenario mirrors one of the most fundamental a...",
    "twitter:image": "https://cdn.hashnode.com/res/hashnode/image/upload/v1756225525421/e62e0a0b-45ba-4980-baf9-947f53e6039a.png",
    "article:published_time": "2025-08-26T16:26:36.659Z",
    "article:author": "https://juancolamendy.hashnode.dev/@juancolamendy"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/fef819142dd2/code/body-000.txt",
      "chars": 808,
      "language": "text"
    },
    {
      "file": "attachments/fef819142dd2/code/body-001.txt",
      "chars": 450,
      "language": "text"
    },
    {
      "file": "attachments/fef819142dd2/code/body-002.txt",
      "chars": 595,
      "language": "text"
    },
    {
      "file": "attachments/fef819142dd2/code/body-003.txt",
      "chars": 349,
      "language": "text"
    },
    {
      "file": "attachments/fef819142dd2/code/body-004.txt",
      "chars": 647,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/res/hashnode/image/upload/v1756225525421/e62e0a0b-45ba-4980-baf9-947f53e6039a.png",
      "bytes": 1953918,
      "reused": true,
      "local_path": "attachments/fef819142dd2/images/000.png",
      "src": "https://cdn.hashnode.com/res/hashnode/image/upload/v1756225525421/e62e0a0b-45ba-4980-baf9-947f53e6039a.png",
      "alt": "Gradient Descent in Deep Learning: A Complete Guide with PyTorch and Keras Examples",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://juancolamendy.hashnode.dev/gradient-descent-in-deep-learning-a-complete-guide-with-pytorch-and-keras-examples.md",
      "label": "View as Markdown",
      "type": "a"
    },
    {
      "href": "https://hashnode.com/@juancolamendy",
      "label": "J",
      "type": "a"
    },
    {
      "href": "https://twitter.com/juancolamendy",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://www.linkedin.com/in/juancolamendy/",
      "label": "",
      "type": "a"
    },
    {
      "href": "http://torch.no/",
      "label": "torch.no",
      "type": "a"
    },
    {
      "href": "https://juancolamendy.hashnode.dev/tag/deep-learning",
      "label": "# deep-learning",
      "type": "a"
    },
    {
      "href": "https://juancolamendy.hashnode.dev/tag/gradient-descent",
      "label": "# gradient-descent",
      "type": "a"
    },
    {
      "href": "https://juancolamendy.hashnode.dev/tag/machine-learning",
      "label": "# machine-learning",
      "type": "a"
    },
    {
      "href": "https://juancolamendy.hashnode.dev/tag/artificial-intelligence",
      "label": "# artificial-intelligence",
      "type": "a"
    },
    {
      "href": "https://juancolamendy.hashnode.dev/tag/keras",
      "label": "# keras",
      "type": "a"
    },
    {
      "href": "https://juancolamendy.hashnode.dev/tag/pytorch",
      "label": "# pytorch",
      "type": "a"
    }
  ],
  "retry_at": "2026-10-03T08:06:06",
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:50:36Z",
  "source_url": "https://juancolamendy.hashnode.dev/gradient-descent-in-deep-learning-a-complete-guide-with-pytorch-and-keras-examples",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 18821,
  "expected_body_images": 1,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/fef819142dd2/pages/source.html",
  "local_html": "attachments/fef819142dd2/pages/article.html",
  "body_base_path": "attachments/fef819142dd2/pages/",
  "body_markdown_path": "attachments/fef819142dd2/pages/article.md"
}