{
  "source_file": "0930",
  "source_platform": "Hashnode",
  "author": "Fadhil Elrizanda",
  "title_listed": "Regularization Techniques to Improve Deep Learning Performance",
  "keywords": "",
  "domain": "机器学习",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://elrizanda.hashnode.dev/regularization-techniques-to-improve-deep-learning-performance",
  "publish_date": "2024-12-27",
  "corpus_stage": "",
  "notes": "",
  "id": "2480ec06e47f",
  "collected_at": "2026-10-02T14:12:12",
  "title": "Regularization Techniques to Improve Deep Learning Performance",
  "body_markdown": "Regularization Techniques to Improve Deep Learning Performance\nUpdated\nMarch 5, 2025\n•\n5\nmin read\n•\nView as Markdown\nF\nFadhil Elrizanda\nOverview\nWhen building and training deep learning models we often facing the problem of overfitting. Overfitting may happen due the model complexity or the model is “memorized” the training dataset instead of learning generalizable patterns. This become the problem because we want the training and validation performance are as close as possible.\nTo tackle this problem we can use regularization method which introduces constraints or penalties to the model’s complexity, encouraging the model to generalize better and reducing its likelihood to overfitting. We will deep dive to this techniques in this section.\nMethods\nThere are many methods that we can use to regularize our models. The regularization techniques seem simple but can affect the models.\n1. Lasso Regularization (L1)\nLASSO stands for Least Absolute Shrinkage and Selection Operator which commonly known as L1 regularization. This technique penalizes the absolute values of the model weight by adding the following formula:\n$$\\mathcal{L}{\\text{total}} = \\mathcal{L}{\\text{original}} + \\lambda \\sum_{i=1}^n |w_i|$$\nWhere :\nLambda (λ) : Regularization strength\nImpact\nL1 regularization promotes sparsity in the model weights, this mean that many weights become exatly zero.\nSparse weights can lead to simpler models that are more interpretable and less prone to overfitting.\nL1 regularization is useful in feature selection tasks, as it effectively eliminates irrelevant feature by setting their weights to zero.\nCommon values between 0.01 to 1 (depends on the dataset)\n2. Ridge Regularization (L2)\nThis technique, ridge regularization known as L2 is similar to L1 but instead penalizes the squared magnitude of the weights. This regularization follow this formula.\n$$\\mathcal{L}{\\text{total}} = \\mathcal{L}{\\text{original}} + \\lambda \\sum_{i=1}^n w_i^2$$\nImpact\nEncourage smaller weights, making the model more stable and less sensitive to noise\nReduces overfitting by preventing any single feature or neuron from dominating the model\nThis technique suitable for reducing overfitting without forcing complete sparsity in the model weights.\nCommon values between 0.01 to 1 (depends on the dataset)\n3. ElasticNet Regularization\nThis regularization combines both L1 and L2 regularizzation into a single loss function.\n$$\\mathcal{L}{\\text{total}} = \\mathcal{L}{\\text{original}} + \\lambda_1 \\sum_{i=1}^n |w_i| + \\lambda_2 \\sum_{i=1}^n w_i^2$$\nWhere :\nλ1 : Coefficient for L1 regularization\nλ2 : Coefficeint for L2 regularization\nImpact\nCombines the benefits of L1 and L2 regularization\nCan handle correlated feature better than L1 but still retains some weights instead of forcing all but one to zero\nThis technique useful when dealing datasets that have many features, especially when some of them are correlate\nCommon values between 0.01 to 1 (depends on the dataset)\n4. Dropout\nDropout is important regularization in neural networks. It work by randomly dropping the neuron (or kill the neuron randomly with probability) in which training epochs. This method encourage the neuron to be robust and not heavily depend on one or particular neuron. The following formula is\n$$y_i = \\begin{cases} 0 & \\text{with probability } p \\\\ \\frac{a_i}{1-p} & \\text{with probability } 1-p \\end{cases}$$\nImpact\nForces the network to learn more robust features by not relying too much on particular neuron\nReduces overfitting significantly\nThis method widely use in deep learning model\nCommon values between 0.3 to 0.5\n5. Early Stopping\nThis technique is simple to prevent overfitting. It monitors the validation loss during training and stops the training when the validation performance starts to degrade (overfitting). The model is trained for fixed number of epochs, but training halts early if the validation loss does not improve (degrade) for a ertain number of epochs which controlled by patience parameter.\nImpacts\nPrevents wasting time and computational resources on overfitting\nOften combined with other regularization method\nCommonly use for training large models\nCommon values between 3 to 10\n6. Data Augmentation\nData augmentation generates additional data by applying transformations to the original dataset, helping to generalize model better. In computer vision, common transformation include:\nRandom Brightness\nRandom Crop\nRandom Flip\nRandom Hue\nImpacts\nSimulates variotion in the data, making the model more robust to changes in input\nReduce overfitting by effectively increasing the dataset size\n7. Batch Normalization\nBatch normalization normalizes the inputs to each layer during training, ensuring that the mean is zero and the variance is one. The normalized output is scaled and shifted using learnable parameter gamma and beta.\n$$\\hat{x} = \\frac{x - \\mu}{\\sqrt{\\sigma^2 + \\epsilon}}, \\quad y = \\gamma \\hat{x} + \\beta$$\nImpacts\nReduces internal covariate shift, speeding up convergence\nActs as regularizer, reducing the need for dropout in some cases\nWidely use in modern architectures.\nϵ is small constant to add variance and γ (Scale) and β (Shift) are learnable parameters.\nSummary\nMethod\nImpact\nUse Case\nL1 Regularization\nPromotes sparsity\nFeature selection and interpretability\nL2 Regularization\nEncourages smaller weights\nStabilizing models, reducing noise sensitivity\nElastic Net\nCombines L1 and L2 benefits\nHandling correlated features\nDropout\nPrevents co-adaptation of neurons\nDeep networks in CV, NLP, etc.\nEarly Stopping\nStops overfitting during training\nTime-efficient training monitoring\nData Augmentation\nIncreases data variability\nComputer vision, NLP\nBatch Normalization\nNormalizes layer inputs, reduces shift\nDeep architectures like ResNet, Inception\nRegularization ensures that deep learning models generalize well, preventing them from overfitting to the training data. By incorporating these techniques, you can build models that perform robustly on unseen datasets.\n#\nmachine-learning\n#\nartificial-intelligence\n#\ncomputer-vision\n#\nregularization\n#\nalgorithms",
  "metadata": {
    "title": "Regularization Techniques to Improve Deep Learning Performance",
    "description": "Overview\n\nWhen building and training deep learning models we often facing the problem of overfitting. Overfitting may happen due the model complexity or the model is “memorized” the training dataset instead of learning generalizable patterns. This be...",
    "keywords": null,
    "author": null,
    "og:title": "Regularization Techniques to Improve Deep Learning Performance",
    "og:description": "Overview\n\nWhen building and training deep learning models we often facing the problem of overfitting. Overfitting may happen due the model complexity or the model is “memorized” the training dataset instead of learning generalizable patterns. This be...",
    "og:url": "https://elrizanda.hashnode.dev/regularization-techniques-to-improve-deep-learning-performance",
    "og:site_name": "elrizanda",
    "og:image": "https://cdn.hashnode.com/res/hashnode/image/upload/v1735340995174/5b51b9ba-8566-4ebe-93d8-7da152942df7.gif",
    "og:image:width": "1200",
    "og:image:height": "630",
    "og:type": "article",
    "twitter:card": "summary_large_image",
    "twitter:title": "Regularization Techniques to Improve Deep Learning Performance",
    "twitter:description": "Overview\n\nWhen building and training deep learning models we often facing the problem of overfitting. Overfitting may happen due the model complexity or the model is “memorized” the training dataset instead of learning generalizable patterns. This be...",
    "twitter:image": "https://cdn.hashnode.com/res/hashnode/image/upload/v1735340995174/5b51b9ba-8566-4ebe-93d8-7da152942df7.gif",
    "article:published_time": "2024-12-27T23:09:00.187Z",
    "article:modified_time": "2025-03-05T06:56:10.401Z",
    "article:author": "https://elrizanda.hashnode.dev/@fadhilelrizanda"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [],
  "images": [
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/res/hashnode/image/upload/v1735340995174/5b51b9ba-8566-4ebe-93d8-7da152942df7.gif",
      "bytes": 381285,
      "reused": true,
      "local_path": "attachments/2480ec06e47f/images/000.gif",
      "src": "https://cdn.hashnode.com/res/hashnode/image/upload/v1735340995174/5b51b9ba-8566-4ebe-93d8-7da152942df7.gif",
      "alt": "Regularization Techniques to Improve Deep Learning Performance",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/res/hashnode/image/upload/v1735340864065/f4d0866b-536f-49fd-a853-f1fddebfc727.gif",
      "bytes": 381285,
      "reused": true,
      "local_path": "attachments/2480ec06e47f/images/002.gif",
      "src": "https://cdn.hashnode.com/res/hashnode/image/upload/v1735340864065/f4d0866b-536f-49fd-a853-f1fddebfc727.gif",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://elrizanda.hashnode.dev/regularization-techniques-to-improve-deep-learning-performance.md",
      "label": "View as Markdown",
      "type": "a"
    },
    {
      "href": "https://hashnode.com/@fadhilelrizanda",
      "label": "F",
      "type": "a"
    },
    {
      "href": "https://www.linkedin.com/in/fadhil-elrizanda/",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://elrizanda.hashnode.dev/tag/machine-learning",
      "label": "# machine-learning",
      "type": "a"
    },
    {
      "href": "https://elrizanda.hashnode.dev/tag/artificial-intelligence",
      "label": "# artificial-intelligence",
      "type": "a"
    },
    {
      "href": "https://elrizanda.hashnode.dev/tag/computer-vision",
      "label": "# computer-vision",
      "type": "a"
    },
    {
      "href": "https://elrizanda.hashnode.dev/tag/regularization",
      "label": "# regularization",
      "type": "a"
    },
    {
      "href": "https://elrizanda.hashnode.dev/tag/algorithms",
      "label": "# algorithms",
      "type": "a"
    }
  ],
  "retry_at": "2026-10-03T08:09:40",
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:51:00Z",
  "source_url": "https://elrizanda.hashnode.dev/regularization-techniques-to-improve-deep-learning-performance",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 6086,
  "expected_body_images": 2,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/2480ec06e47f/pages/source.html",
  "local_html": "attachments/2480ec06e47f/pages/article.html",
  "body_base_path": "attachments/2480ec06e47f/pages/",
  "body_markdown_path": "attachments/2480ec06e47f/pages/article.md"
}