{
  "source_file": "0930",
  "source_platform": "研究机构",
  "author": "Dive into Deep Learning",
  "title_listed": "Optimization Algorithms（章节）",
  "keywords": "SGD,Adam,优化器",
  "domain": "机器学习",
  "content_type": "教程/书籍",
  "is_paid_flag": "否",
  "url": "https://www.d2l.ai/chapter_optimization/index.html",
  "publish_date": "持续更新",
  "corpus_stage": "机器学习",
  "notes": "优化算法章节",
  "id": "078281d0595f",
  "collected_at": "2026-10-02T16:07:43",
  "title": "12. Optimization Algorithms — Dive into Deep Learning 1.0.3 documentation",
  "body_markdown": "Table Of Contents\n12.\nOptimization Algorithms\n¶\nIf you read the book in sequence up to this point you already used a\nnumber of optimization algorithms to train deep learning models. They\nwere the tools that allowed us to continue updating model parameters and\nto minimize the value of the loss function, as evaluated on the training\nset. Indeed, anyone content with treating optimization as a black box\ndevice to minimize objective functions in a simple setting might well\ncontent oneself with the knowledge that there exists an array of\nincantations of such a procedure (with names such as “SGD” and “Adam”).\nTo do well, however, some deeper knowledge is required. Optimization\nalgorithms are important for deep learning. On the one hand, training a\ncomplex deep learning model can take hours, days, or even weeks. The\nperformance of the optimization algorithm directly affects the model’s\ntraining efficiency. On the other hand, understanding the principles of\ndifferent optimization algorithms and the role of their hyperparameters\nwill enable us to tune the hyperparameters in a targeted manner to\nimprove the performance of deep learning models.\nIn this chapter, we explore common deep learning optimization algorithms\nin depth. Almost all optimization problems arising in deep learning are\nnonconvex\n. Nonetheless, the design and analysis of algorithms in the\ncontext of\nconvex\nproblems have proven to be very instructive. It is\nfor that reason that this chapter includes a primer on convex\noptimization and the proof for a very simple stochastic gradient descent\nalgorithm on a convex objective function.\n12.1. Optimization and Deep Learning\n12.1.1. Goal of Optimization\n12.1.2. Optimization Challenges in Deep Learning\n12.1.3. Summary\n12.1.4. Exercises\n12.2. Convexity\n12.2.1. Definitions\n12.2.2. Properties\n12.2.3. Constraints\n12.2.4. Summary\n12.2.5. Exercises\n12.3. Gradient Descent\n12.3.1. One-Dimensional Gradient Descent\n12.3.2. Multivariate Gradient Descent\n12.3.3. Adaptive Methods\n12.3.4. Summary\n12.3.5. Exercises\n12.4. Stochastic Gradient Descent\n12.4.1. Stochastic Gradient Updates\n12.4.2. Dynamic Learning Rate\n12.4.3. Convergence Analysis for Convex Objectives\n12.4.4. Stochastic Gradients and Finite Samples\n12.4.5. Summary\n12.4.6. Exercises\n12.5. Minibatch Stochastic Gradient Descent\n12.5.1. Vectorization and Caches\n12.5.2. Minibatches\n12.5.3. Reading the Dataset\n12.5.4. Implementation from Scratch\n12.5.5. Concise Implementation\n12.5.6. Summary\n12.5.7. Exercises\n12.6. Momentum\n12.6.1. Basics\n12.6.2. Practical Experiments\n12.6.3. Theoretical Analysis\n12.6.4. Summary\n12.6.5. Exercises\n12.7. Adagrad\n12.7.1. Sparse Features and Learning Rates\n12.7.2. Preconditioning\n12.7.3. The Algorithm\n12.7.4. Implementation from Scratch\n12.7.5. Concise Implementation\n12.7.6. Summary\n12.7.7. Exercises\n12.8. RMSProp\n12.8.1. The Algorithm\n12.8.2. Implementation from Scratch\n12.8.3. Concise Implementation\n12.8.4. Summary\n12.8.5. Exercises\n12.9. Adadelta\n12.9.1. The Algorithm\n12.9.2. Implementation\n12.9.3. Summary\n12.9.4. Exercises\n12.10. Adam\n12.10.1. The Algorithm\n12.10.2. Implementation\n12.10.3. Yogi\n12.10.4. Summary\n12.10.5. Exercises\n12.11. Learning Rate Scheduling\n12.11.1. Toy Problem\n12.11.2. Schedulers\n12.11.3. Policies\n12.11.4. Summary\n12.11.5. Exercises\nPrevious\n11.9. Large-Scale Pretraining with Transformers\nNext\n12.1. Optimization and Deep Learning",
  "metadata": {
    "title": "12. Optimization Algorithms — Dive into Deep Learning 1.0.3 documentation",
    "description": null,
    "keywords": null,
    "author": null
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [],
  "images": [
    {
      "status": "ok",
      "url": "https://www.d2l.ai/_static/logo-with-text.png",
      "bytes": 87175,
      "reused": false,
      "local_path": "attachments/078281d0595f/images/body-509f124ba61493e64845.png",
      "src": "https://www.d2l.ai/_static/logo-with-text.png",
      "alt": "Dive into Deep Learning",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://www.d2l.ai/index.html",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://www.d2l.ai/chapter_optimization/optimization-intro.html",
      "label": "12.1. Optimization and Deep Learning",
      "type": "a"
    },
    {
      "href": "https://www.d2l.ai/chapter_optimization/convexity.html",
      "label": "12.2. Convexity",
      "type": "a"
    },
    {
      "href": "https://www.d2l.ai/chapter_optimization/gd.html",
      "label": "12.3. Gradient Descent",
      "type": "a"
    },
    {
      "href": "https://www.d2l.ai/chapter_optimization/sgd.html",
      "label": "12.4. Stochastic Gradient Descent",
      "type": "a"
    },
    {
      "href": "https://www.d2l.ai/chapter_optimization/minibatch-sgd.html",
      "label": "12.5. Minibatch Stochastic Gradient Descent",
      "type": "a"
    },
    {
      "href": "https://www.d2l.ai/chapter_optimization/momentum.html",
      "label": "12.6. Momentum",
      "type": "a"
    },
    {
      "href": "https://www.d2l.ai/chapter_optimization/adagrad.html",
      "label": "12.7. Adagrad",
      "type": "a"
    },
    {
      "href": "https://www.d2l.ai/chapter_optimization/rmsprop.html",
      "label": "12.8. RMSProp",
      "type": "a"
    },
    {
      "href": "https://www.d2l.ai/chapter_optimization/adadelta.html",
      "label": "12.9. Adadelta",
      "type": "a"
    },
    {
      "href": "https://www.d2l.ai/chapter_optimization/adam.html",
      "label": "12.10. Adam",
      "type": "a"
    },
    {
      "href": "https://www.d2l.ai/chapter_optimization/lr-scheduler.html",
      "label": "12.11. Learning Rate Scheduling",
      "type": "a"
    },
    {
      "href": "https://www.d2l.ai/chapter_attention-mechanisms-and-transformers/large-pretraining-transformers.html",
      "label": "Previous 11.9. Large-Scale Pretraining with Transformers",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:53:48Z",
  "source_url": "https://www.d2l.ai/chapter_optimization/index.html",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 3382,
  "expected_body_images": 1,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/078281d0595f/pages/source.html",
  "local_html": "attachments/078281d0595f/pages/article.html",
  "body_base_path": "attachments/078281d0595f/pages/",
  "body_markdown_path": "attachments/078281d0595f/pages/article.md"
}