{
  "source_file": "0930",
  "source_platform": "Hashnode",
  "author": "Fadhil Elrizanda",
  "title_listed": "Diving into Transformer : Attention Is All You Need!",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://elrizanda.hashnode.dev/diving-into-transformer-attention-is-all-you-need",
  "publish_date": "2025-03-10",
  "corpus_stage": "",
  "notes": "",
  "id": "dbf34b78951e",
  "collected_at": "2026-10-02T14:12:10",
  "title": "Diving into Transformer : Attention Is All You Need!",
  "body_markdown": "Diving into Transformer : Attention Is All You Need!\nUpdated\nMarch 10, 2025\n•\n6\nmin read\n•\nView as Markdown\nF\nFadhil Elrizanda\nTransformer is a deep neural network architecture that introduced in “attention is all you need”. It is designed for sequence to sequence tasks. Particularly, excelling in natural language processing (NLP). It eliminates RNN and CNN in favor of self attention and feedforward layers. Transformer become one of the state-of-the-art in deep learning architecture.\nSource :\nhttps://arxiv.org/abs/1706.03762\n1. Key Component of Transformer\n1. Encoder Architecture\nTransformer encoder extracts contextual meaning using attention mechanism. This encoder consist of stacked layers. Each encoded layer consist of:\nMulti head self attention\nFeedforward network (FFN)\nAdd and Layer Normalization\n2. Decoder Architecture\nTransformer decoder generates token in sequences, on token at a time. Each decoder layer has the same components as the encoder, but with an additional masker multi head self attention layer to prevent future information leakage. This layer also repeated multiple time. Decoder component consist of :\nMasked Multi Head Self attention\nEncoder Decoder Multi head attention\nFeedforward Network\nAdd and Layer Normalization\nComparison Transformer Encoder vs Decoder\nComponent\nEncoder\nDecoder\nSelf-Attention\nFull\nMasked\nEncoder-Decoder Attention\n❌\n✅\nFeed-Forward Network\n✅\n✅\nAuto-Regressive\n❌\n✅\nFinal Softmax Output\n❌\n✅\n2. Step by step data flow in the Transformer\n1. Input Tokenization\nThe raw text is converted into tokens (sub words or words) using Byte Pair Encoding or WordPiece in GPT)\nExample:\nInput sentence:\n\"I love AI\"\nTokenized:\n[\n\"I\"\n,\n\"love\"\n,\n\"AI\"\n]\nThen each token mapped to vocabulary indices:\n[\n45\n,\n987\n,\n210\n]\n2. Token Embedding\nEach token ID is converted into high-dimensional vector using an embedding matrix. The goal of this step is to convert token IDs into dense matrix to capture meaning.\n$$X = E \\cdot I$$\nWhere :\n$$E \\in \\mathbb{R}^{V \\times d} \\text{ is the trainable embedding matrix}$$\nis the trainable embedding matrix\nI\nis the input token index\nd\nis the embedding dimension\nExample :\n[\n101\n] → [\n0.1\n,\n0.3\n,\n-0.2\n,\n0.7\n, ...]\n[\n2345\n] → [\n0.25\n,\n-0.12\n,\n0.87\n,\n0.56\n, ...]\n3. Positional Embedding\nIn transformer architecture, it doesn’t process each word sequentially, so it need to add positional information. The positional information is generated using a sine and cosine function to generates unique position encoding. Positional encoding formula\n$$PE_{(pos, 2i)} = \\sin \\left( \\frac{pos}{10000^{\\frac{2i}{d}}} \\right)$$\n$$PE_{(pos, 2i+1)} = \\cos \\left( \\frac{pos}{10000^{\\frac{2i}{d}}} \\right)$$\nWhere :\npos\nis the position in the sequence\nd\nis the embedding dimension\ni\nis the dimension index\nThe positional encoding is added to the word embedding. The formula becomes :\n$$\\text{Final Representation} = \\text{Embedding} + \\text{Positional Encoding}$$\n4. Attention Mechanism Encoder\nAfter all data are converted into final representation, it can be passing to the transformer encoder. Transformer encoder used to capture the contextual meaning. Each transformer encoder block consist of:\nMulti head Self Attention\nAdd and Layer Normalization\nFeedforward Network\nAdd and Layer Normalization\n4.1 Compute Queries (Q), Keys (K), and Value (V)\nThis step converts the embedding using learnable metrics\n$$W_{Q}, \\quad W_{K}, \\quad W_{V}$$\n$$Q = XW_{Q}, \\quad K = XW_{K}, \\quad V = XW_{V}$$\n4.2 Compute Attention Scores\nThe attention scores are computed using attention weight using dot product attention mechanism\n$$\\text{Attention Score} = Q K^{T}$$\nAfter computing the attention score, we need to normalize it using softmax\n4.3 Compute Weighted Sum of Values\nThis step is multiplying the attention score with Value (V) vectors\n$$\\text{Final Attention Output} = \\sum (\\text{attention weight} \\times V)$$\nThis step produce word with contextual aware information\n5. Feedforward Layers\nThis layer is applying additional processing to attention output. This layer often used in modern deep learning architecture. Each attention output is passed through a fully feedforward network\n$$\\text{Output} = \\text{ReLU}(W_{1} + b_{1}) W_{2} + b_{2}$$\n6. Repeating transformer layers\nStep 4-5 are repeated N time (e.g 12 layers in BERT, 96 in GPT-3)\n7. Transformer Decoder (for GPT-like Model)\nTransformer decoder is responsible to generating output sequence one token at time. The decoder using previously generated token and encoder output. It consist of multiple layers (e.g 6 layers in the original transformer, 96 in GPT) and follows up with auto regressive generation process. Each decoder layer has three main component:\nMasked Multi Head Self attention (Decoder attends to past generated token)\nEncoder - Decoder Multi Head Attention (Decoder attends to encoder outputs)\nFeedforward network (Process token reprsesentaion)\n7.1 Initialize the decoder with special token\nThis process are same like the encoder process\n7.2 Compute token embeddings and positional encodings\nThis process are same like the encoder process\n7.3 Masked Multi Head Self attention\nAt this step, the decoder applies self attention to previously generated token. A casual mask is implemented to ensure that a token at position t cannot attend to future token t + 1, t +2, ….\nThere are similar step like computing queries, keys, and values then computing attention scores just like in encoder.\nApplying casual masking\nTo prevent the model from looking ahead, masking is implemented to mask ou future position by setting their attention score to - infinity, ensuring only previous token influence prediction.\n7.4 Encoder-decoder multi head attention\nThis step allow the decoder to attend the encoders output. This step involve :\nQueries from the decoder\nKeys (K) and Values (V) from encoder otpt\nThe formula remain the same\nThis step align the generated text with the input sentence\n\"I love AI\"\n→ Encoder → Context Representation\n\"J' \"\n→ Decoder attends to encoded\n\"I\"\n\"I love AI\" → Encoder → Context Representation \"J' \" → Decoder attends to encoded \"I\"\n7.5 Applying Feed Forward Network (FFN), Residual Network and Normalization\nThis process involve traditional FFN, residual network and layer normalization in modern deep learning architecture.\n8. Final Output Prediction\nThis step is to convert final vector into words\nApply a linear transformation to map embedding to vocabulary logits\nApply softmax to get probabilities\nSelect the highest probability token\nLogits: [\n0.1\n,\n0.6\n,\n0.2\n,\n0.9\n]\nSoftmax Probabilities: [\n10\n%,\n60\n%,\n20\n%,\n90\n%]\nPredicted word:\n\"fast\"\n9. Post processing\nThis step is to convert model’s prediction into human readible sentences\nconvert token IDs back to words\nHandle punctuation, capitalization, and formatting\nIf generating text, repeat steps until reaching an end token ([EOS])\nExample\n\"The cat runs fast.\"\nSummary of the Data Flow:\nStep\nProcess\nPurpose\n1\nTokenization\nConvert text to numerical tokens\n2\nEmbedding Layer\nMap tokens to dense vectors\n3\nPositional Encoding\nAdd word order information\n4\nSelf-Attention\nCompute relationships between words\n5\nFeedforward Layer\nProcess attention outputs\n6\nRepeat Transformer Layers\nRefine representations across multiple layers\n7\nDecoder (Optional)\nGenerate output step by step\n8\nSoftmax & Word Selection\nPredict final words\n9\nPost-Processing\nConvert back to readable text\n#\ndeep-learning\n#\ntransformers\n#\npython\n#\nnatural-language-processing\n#\nartificial-intelligence",
  "metadata": {
    "title": "Diving into Transformer : Attention Is All You Need!",
    "description": "Transformer is a deep neural network architecture that introduced in “attention is all you need”. It is designed for sequence to sequence tasks.",
    "keywords": null,
    "author": null,
    "og:title": "Diving into Transformer : Attention Is All You Need!",
    "og:description": "Transformer is a deep neural network architecture that introduced in “attention is all you need”. It is designed for sequence to sequence tasks.",
    "og:url": "https://elrizanda.hashnode.dev/diving-into-transformer-attention-is-all-you-need",
    "og:site_name": "elrizanda",
    "og:image": "https://cdn.hashnode.com/res/hashnode/image/upload/v1741580235464/2d712d06-1c25-4cc1-a2f5-0ab8d7b79b79.gif",
    "og:image:width": "1200",
    "og:image:height": "630",
    "og:type": "article",
    "twitter:card": "summary_large_image",
    "twitter:title": "Diving into Transformer : Attention Is All You Need!",
    "twitter:description": "Transformer is a deep neural network architecture that introduced in “attention is all you need”. It is designed for sequence to sequence tasks.",
    "twitter:image": "https://cdn.hashnode.com/res/hashnode/image/upload/v1741580235464/2d712d06-1c25-4cc1-a2f5-0ab8d7b79b79.gif",
    "article:published_time": "2025-03-10T04:21:06.959Z",
    "article:modified_time": "2025-03-10T04:33:11.639Z",
    "article:author": "https://elrizanda.hashnode.dev/@fadhilelrizanda"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/dbf34b78951e/code/body-000.txt",
      "chars": 12,
      "language": "text"
    },
    {
      "file": "attachments/dbf34b78951e/code/body-001.txt",
      "chars": 20,
      "language": "text"
    },
    {
      "file": "attachments/dbf34b78951e/code/body-002.txt",
      "chars": 15,
      "language": "text"
    },
    {
      "file": "attachments/dbf34b78951e/code/body-003.txt",
      "chars": 75,
      "language": "text"
    },
    {
      "file": "attachments/dbf34b78951e/code/body-004.txt",
      "chars": 86,
      "language": "text"
    },
    {
      "file": "attachments/dbf34b78951e/code/body-005.txt",
      "chars": 96,
      "language": "text"
    },
    {
      "file": "attachments/dbf34b78951e/code/body-006.txt",
      "chars": 21,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/res/hashnode/image/upload/v1741580235464/2d712d06-1c25-4cc1-a2f5-0ab8d7b79b79.gif",
      "bytes": 162928,
      "reused": true,
      "local_path": "attachments/dbf34b78951e/images/000.gif",
      "src": "https://cdn.hashnode.com/res/hashnode/image/upload/v1741580235464/2d712d06-1c25-4cc1-a2f5-0ab8d7b79b79.gif",
      "alt": "Diving into Transformer : Attention Is All You Need!",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/res/hashnode/image/upload/v1741580269372/a46c4c0f-2be6-4248-b7bd-cae797311ba4.gif",
      "bytes": 162928,
      "reused": true,
      "local_path": "attachments/dbf34b78951e/images/002.gif",
      "src": "https://cdn.hashnode.com/res/hashnode/image/upload/v1741580269372/a46c4c0f-2be6-4248-b7bd-cae797311ba4.gif",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://elrizanda.hashnode.dev/diving-into-transformer-attention-is-all-you-need.md",
      "label": "View as Markdown",
      "type": "a"
    },
    {
      "href": "https://hashnode.com/@fadhilelrizanda",
      "label": "F",
      "type": "a"
    },
    {
      "href": "https://www.linkedin.com/in/fadhil-elrizanda/",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1706.03762",
      "label": "https://arxiv.org/abs/1706.03762",
      "type": "a"
    },
    {
      "href": "https://elrizanda.hashnode.dev/tag/deep-learning",
      "label": "# deep-learning",
      "type": "a"
    },
    {
      "href": "https://elrizanda.hashnode.dev/tag/transformers",
      "label": "# transformers",
      "type": "a"
    },
    {
      "href": "https://elrizanda.hashnode.dev/tag/python",
      "label": "# python",
      "type": "a"
    },
    {
      "href": "https://elrizanda.hashnode.dev/tag/natural-language-processing",
      "label": "# natural-language-processing",
      "type": "a"
    },
    {
      "href": "https://elrizanda.hashnode.dev/tag/artificial-intelligence",
      "label": "# artificial-intelligence",
      "type": "a"
    }
  ],
  "retry_at": "2026-10-03T08:09:34",
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:51:00Z",
  "source_url": "https://elrizanda.hashnode.dev/diving-into-transformer-attention-is-all-you-need",
  "papers": [
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1706.03762",
      "bytes": 2215244,
      "reused": true,
      "local_path": "papers/repair/c3ab01e47e614fe3cf33.pdf"
    }
  ],
  "actual_video_files": [],
  "body_chars": 7470,
  "expected_body_images": 2,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/dbf34b78951e/pages/source.html",
  "local_html": "attachments/dbf34b78951e/pages/article.html",
  "body_base_path": "attachments/dbf34b78951e/pages/",
  "body_markdown_path": "attachments/dbf34b78951e/pages/article.md"
}