{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "Priyesh Dave",
  "title_listed": "Day 25: Bahdanau vs. Luong Attention",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "技术教程",
  "is_paid_flag": "否",
  "url": "https://dev.to/priyeshdave6/day-25-bahdanau-vs-luong-attention-3fin",
  "publish_date": "2026-09-26",
  "corpus_stage": "",
  "notes": "",
  "id": "975f0d38b3de",
  "collected_at": "2026-10-02T13:49:09",
  "title": "Day 25: Bahdanau vs. Luong Attention - DEV Community",
  "body_markdown": "What Is Attention in Sequence Models?\nAttention is a technique that allows neural networks to focus on the most relevant parts of their input when generating each output, instead of giving every input element equal weight.\nThink of translating a sentence from French to English. When you produce each English word, your focus shifts to the French words most closely related to what you’re writing—like focusing on verbs when translating a verb. You don’t need to keep the entire sentence in mind at every step.\nIn natural language processing (NLP), attention enables a model to mimic this selective focus. Rather than squeezing a whole sentence into a single summary vector, attention lets the model “look back” and weigh parts of the input differently for each output word.\nThe Problem: Seq2Seq Bottlenecks\nBasic sequence-to-sequence (seq2seq) models use two parts:\nAn\nencoder\nturns an input sequence (like a sentence) into a single summary vector.\nA\ndecoder\nconverts that vector into an output sequence (such as a translation).\nThis approach works for short sequences but stumbles on longer ones. The model tries to cram all meaning into one vector, losing details. For long sentences, important words fade. The model may lose track of “who did what to whom.”\nAttention allows the decoder to refer directly to any part of the encoder’s output at each output step. This sidesteps the compression bottleneck.\nBahdanau Attention: The First \"Soft\" Attention\nBahdanau attention, also called\nadditive attention\n, was the first widely-used attention mechanism in NLP. Its key idea: for each word the decoder produces, the model decides how much to focus on each word from the input.\nSuppose the encoder processes a sentence, producing a\nhidden state\nfor each word:\nh₁, h₂, ..., hₙ\n(each h is a vector summarizing a word’s context).\nThe decoder also maintains a\nhidden state\nat each output step, labeled\nsₜ\n.\nBahdanau attention asks: for the current output word, which encoder states are most relevant? It answers this by assigning an\nalignment score\nto each encoder state, reflecting how well it matches the decoder’s current state.\nUnlike a simple similarity (like dot product), Bahdanau attention uses a small neural network (a single feedforward layer with tanh activation) to compute each score.\nThese scores, once normalized (adding up to 1 using softmax), become weights. The model computes a\nweighted average\nof all encoder hidden states using these weights—a new\ncontext vector\n. This vector provides the decoder with focused information from the encoder, tailored to each output step.\nBahdanau Attention Step-by-Step\nAt each output step, Bahdanau attention goes through:\nScore Calculation:\nFor every encoder hidden state\nhᵢ\n, combine it with the decoder state\nsₜ\nto get a score:\nscoreᵢ = vᵀ tanh(W₁ hᵢ + W₂ sₜ)\nEnter fullscreen mode\nExit fullscreen mode\nHere,\nW₁\n,\nW₂\n, and\nv\nare trainable weights.\ntanh\nis a nonlinear function that limits outputs to a range between -1 and 1.\nSoftmax Normalization:\nConvert all the raw scores into attention weights (probabilities), ensuring they sum to 1:\nαᵢ = softmax(scoreᵢ)\nEnter fullscreen mode\nExit fullscreen mode\nContext Vector Computation:\nUse attention weights to take the weighted average of encoder hidden states:\ncontext = Σ (αᵢ * hᵢ)\nEnter fullscreen mode\nExit fullscreen mode\nThe context vector is then fed to the decoder to help generate the next output word.\nLuong Attention: A Simpler, Faster Variant\nLuong attention, also called\nmultiplicative attention\n, simplifies and speeds up the scoring step. It makes two main changes:\nWhen attention is applied:\nBahdanau applies attention before the decoder updates its hidden state; Luong applies it after. This changes the flow of information but not the core idea.\nHow scores are computed:\nLuong offers simpler score functions. Instead of a feedforward network, it uses one of:\nDot product:\nDirectly measures similarity between encoder and decoder hidden states.\nGeneral:\nLike dot, but introducing a learned weight matrix.\nIn notation:\nscoreᵢ\n=\nsₜᵀ\nW\nhᵢ\n# general\nscoreᵢ\n=\nsₜᵀ\nhᵢ\n# dot\nEnter fullscreen mode\nExit fullscreen mode\nLike Bahdanau, the softmax normalization and context vector calculation remain the same.\nComparing Bahdanau and Luong Attention\nFeature\nBahdanau (Additive)\nLuong (Multiplicative)\nScoring\nNeural net with tanh & weights\nDot/general (matrix multiply)\nComputation\nBefore decoder update\nAfter decoder update\nEfficiency\nMore parameters, slower\nFewer parameters, faster\nCommon Usage\nEarly NLP sequence tasks\nLater, scalable models\nChoosing an approach:\nBahdanau’s method is more flexible for complex tasks and small datasets. Luong’s version is faster and better for large batches or long inputs.\nCode Example: Bahdanau and Luong Attention in PyTorch\nBelow is a minimal example showing Bahdanau and Luong attention side by side. You can see how scores and context vectors are computed.\nimport\ntorch\nimport\ntorch.nn\nas\nnn\nimport\ntorch.nn.functional\nas\nF\n# Dummy data: batch size 1, 5 input steps, hidden size 4\nencoder_outputs\n=\ntorch\n.\nrandn\n(\n1\n,\n5\n,\n4\n)\n# (batch, seq_len, hidden)\ndecoder_hidden\n=\ntorch\n.\nrandn\n(\n1\n,\n4\n)\n# (batch, hidden)\n### Bahdanau Attention (additive) ###\nclass\nBahdanauAttention\n(\nnn\n.\nModule\n):\ndef\n__init__\n(\nself\n,\nhidden_size\n):\nsuper\n().\n__init__\n()\nself\n.\nW1\n=\nnn\n.\nLinear\n(\nhidden_size\n,\nhidden_size\n)\nself\n.\nW2\n=\nnn\n.\nLinear\n(\nhidden_size\n,\nhidden_size\n)\nself\n.\nv\n=\nnn\n.\nLinear\n(\nhidden_size\n,\n1\n)\ndef\nforward\n(\nself\n,\nencoder_outputs\n,\ndecoder_hidden\n):\nseq_len\n=\nencoder_outputs\n.\nsize\n(\n1\n)\n# Expand decoder hidden to shape (batch, seq_len, hidden)\ndec\n=\ndecoder_hidden\n.\nunsqueeze\n(\n1\n).\nrepeat\n(\n1\n,\nseq_len\n,\n1\n)\nscore\n=\nself\n.\nv\n(\ntorch\n.\ntanh\n(\nself\n.\nW1\n(\nencoder_outputs\n)\n+\nself\n.\nW2\n(\ndec\n))\n).\nsqueeze\n(\n-\n1\n)\n# (batch, seq_len)\nattn_weights\n=\nF\n.\nsoftmax\n(\nscore\n,\ndim\n=\n1\n)\ncontext\n=\ntorch\n.\nbmm\n(\nattn_weights\n.\nunsqueeze\n(\n1\n),\nencoder_outputs\n).\nsqueeze\n(\n1\n)\nreturn\nattn_weights\n,\ncontext\nbahdanau\n=\nBahdanauAttention\n(\nhidden_size\n=\n4\n)\nbahdanau_weights\n,\nbahdanau_context\n=\nbahdanau\n(\nencoder_outputs\n,\ndecoder_hidden\n)\nprint\n(\n\"\nBahdanau attention weights:\n\"\n,\nbahdanau_weights\n)\nprint\n(\n\"\nBahdanau context vector:\n\"\n,\nbahdanau_context\n)\n### Luong Attention (dot) ###\nclass\nLuongAttention\n(\nnn\n.\nModule\n):\ndef\n__init__\n(\nself\n):\nsuper\n().\n__init__\n()\ndef\nforward\n(\nself\n,\nencoder_outputs\n,\ndecoder_hidden\n):\n# encoder_outputs: (batch, seq_len, hidden)\n# decoder_hidden: (batch, hidden)\nattn_weights\n=\ntorch\n.\nbmm\n(\nencoder_outputs\n,\ndecoder_hidden\n.\nunsqueeze\n(\n-\n1\n)\n).\nsqueeze\n(\n-\n1\n)\n# (batch, seq_len)\nattn_weights\n=\nF\n.\nsoftmax\n(\nattn_weights\n,\ndim\n=\n1\n)\ncontext\n=\ntorch\n.\nbmm\n(\nattn_weights\n.\nunsqueeze\n(\n1\n),\nencoder_outputs\n).\nsqueeze\n(\n1\n)\nreturn\nattn_weights\n,\ncontext\nluong\n=\nLuongAttention\n()\nluong_weights\n,\nluong_context\n=\nluong\n(\nencoder_outputs\n,\ndecoder_hidden\n)\nprint\n(\n\"\nLuong attention weights:\n\"\n,\nluong_weights\n)\nprint\n(\n\"\nLuong context vector:\n\"\n,\nluong_context\n)\nEnter fullscreen mode\nExit fullscreen mode\nNotice the difference:\nBahdanau attention uses a small neural network for scoring, which gives it extra flexibility at a computational cost.\nLuong attention (dot version) compares encoder and decoder states directly with a dot product—no extra parameters, so it's faster but less expressive.\nTry adjusting the dummy data and see how attention weights change as the decoder state moves. This testbed makes the core ideas behind sequence model attention visible and tangible.\nKey Takeaways\nAttention lets sequence models focus on relevant input parts for each output step.\nBahdanau attention computes alignment scores using a small neural network, adding expressiveness.\nLuong attention uses simpler dot products for scoring, making it faster and more efficient.\nBoth attention mechanisms produce context vectors as weighted averages of encoder outputs.\nChoosing between Bahdanau and Luong depends on dataset size, task complexity, and efficiency needs.\nTry It Yourself\nUsing the following numpy arrays for encoder hidden states and a decoder hidden state, manually calculate: (1) the Bahdanau alignment scores and attention weights, (2) the Luong dot-product attention scores and weights, and (3) the resulting context vector in each case. Write your computations as explicit arithmetic or code, and compare how the two methods distribute attention.\nFurther Resources\n🎥\nAttention Mechanism (Bahdanau Attention & Luong Attention) | Deep Learning\n📄\nDifferences Between Luong Attention and Bahdanau Attention | Baeldung\n📄\nBahdanau vs Luong Attention: Which One Should You Actually Use? (Spoiler: Luong)\n📄\nEffective Approaches to Attention-based Neural Machine Translation (Luong et al., 2015)\nComing up on Day 26:\n\"Attention Is All You Need\": The Paper That Changed Everything",
  "metadata": {
    "title": "Day 25: Bahdanau vs. Luong Attention - DEV Community",
    "description": "What Is Attention in Sequence Models?   Attention is a technique that allows neural networks... Tagged with ai, nlp, genai, 100daysofcode.",
    "keywords": "ai, nlp, genai, 100daysofcode, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/priyeshdave6/day-25-bahdanau-vs-luong-attention-3fin",
    "og:title": "Day 25: Bahdanau vs. Luong Attention",
    "og:description": "What Is Attention in Sequence Models?   Attention is a technique that allows neural networks...",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fynfe7g4hg6xv27bn68gg.png",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@",
    "twitter:title": "Day 25: Bahdanau vs. Luong Attention",
    "twitter:description": "What Is Attention in Sequence Models?   Attention is a technique that allows neural networks...",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fynfe7g4hg6xv27bn68gg.png"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/975f0d38b3de/code/body-000.txt",
      "chars": 35,
      "language": "text"
    },
    {
      "file": "attachments/975f0d38b3de/code/body-001.txt",
      "chars": 24,
      "language": "text"
    },
    {
      "file": "attachments/975f0d38b3de/code/body-002.txt",
      "chars": 25,
      "language": "text"
    },
    {
      "file": "attachments/975f0d38b3de/code/body-003.txt",
      "chars": 62,
      "language": "text"
    },
    {
      "file": "attachments/975f0d38b3de/code/body-004.txt",
      "chars": 2066,
      "language": "text"
    }
  ],
  "images": [],
  "audio": [],
  "videos": [
    {
      "url": "https://www.youtube.com/watch?v=uaDIK9mYa1U",
      "status": "ok",
      "local_path": "attachments/975f0d38b3de/videos/uaDIK9mYa1U.mp4",
      "bytes": 119834163,
      "valid": true,
      "height": 1080,
      "width": 1920,
      "duration": "2854.361000"
    }
  ],
  "links": [
    {
      "href": "https://www.youtube.com/watch?v=uaDIK9mYa1U",
      "label": "Attention Mechanism (Bahdanau Attention & Luong Attention) | Deep Learning",
      "type": "a"
    },
    {
      "href": "https://www.baeldung.com/cs/attention-luong-vs-bahdanau",
      "label": "Differences Between Luong Attention and Bahdanau Attention | Baeldung",
      "type": "a"
    },
    {
      "href": "https://www.sotaaz.com/post/attention-mechanism-implementation-en",
      "label": "Bahdanau vs Luong Attention: Which One Should You Actually Use? (Spoiler: Luong)",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1508.04025",
      "label": "Effective Approaches to Attention-based Neural Machine Translation (Luong et al., 2015)",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:48:04Z",
  "source_url": "https://dev.to/priyeshdave6/day-25-bahdanau-vs-luong-attention-3fin",
  "papers": [
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1508.04025",
      "bytes": 249825,
      "reused": false,
      "local_path": "papers/repair/94d3d49016ad40753c67.pdf"
    }
  ],
  "actual_video_files": [
    {
      "local_path": "attachments/975f0d38b3de/videos/uaDIK9mYa1U.mp4",
      "bytes": 119834163,
      "valid": true,
      "height": 1080,
      "width": 1920,
      "duration": "2854.361000",
      "status": "ok"
    }
  ],
  "body_chars": 8780,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/975f0d38b3de/pages/source.html",
  "local_html": "attachments/975f0d38b3de/pages/article.html",
  "body_base_path": "attachments/975f0d38b3de/pages/",
  "body_markdown_path": "attachments/975f0d38b3de/pages/article.md"
}