{
  "source_file": "0930",
  "source_platform": "Substack",
  "author": "Seattle Data Guy Ben Rogojan",
  "title_listed": "The Complete Guide to Data Pipelines",
  "keywords": "数据管道,ETL,工程",
  "domain": "数据处理",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://seattledataguy.substack.com/p/complete-guide-data-pipelines",
  "publish_date": "2023-09",
  "corpus_stage": "数据处理",
  "notes": "数据管道完整指南",
  "id": "178d38f5da52",
  "collected_at": "2026-10-01T20:48:35",
  "title": "SeattleDataGuy’s Newsletter | Substack",
  "body_markdown": "[![SeattleDataGuy’s Newsletter](https://substackcdn.com/image/fetch/$s_!fov7!,w_40,h_40,c_fill,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2Fea9135cc-f9d6-4856-8596-2ca9a1655cb6_256x256.png)](https://seattledataguy.substack.com/)\n# [SeattleDataGuy’s Newsletter](https://seattledataguy.substack.com/)\nSubscribeSign in\n[Home](https://seattledataguy.substack.com/)\n[Notes](https://seattledataguy.substack.com/notes)\n[Videos](https://www.youtube.com/@SeattleDataGuy/videos)\n[🚀 Learning Tracks 🚀](https://seattledataguy.substack.com/p/learning-tracks-for-paid-members)\n[Webinars And Deep Dives](https://seattledataguy.substack.com/p/upcoming-data-events)\n[Archive](https://seattledataguy.substack.com/archive)\n[Leaderboard](https://seattledataguy.substack.com/leaderboard)\n[About](https://seattledataguy.substack.com/about)\n## Page not found\nLatestTopDiscussions\n[Do You Actually Need Real-Time Data?](https://seattledataguy.substack.com/p/do-you-actually-need-real-time-data)\n[Why “real-time” is often a business requirement that hasn’t been defined yet](https://seattledataguy.substack.com/p/do-you-actually-need-real-time-data)\nSep 25 • [SeattleDataGuy](https://substack.com/@seattledataguy)\n37\n8\n4\n![](../images/013.jpeg)\n[Engineers Are Obsessed With The How And Often Forget Why?](https://seattledataguy.substack.com/p/engineers-are-obsessed-with-the-how)\n[Why data teams need to spend less time debating the stack and more time questioning the work](https://seattledataguy.substack.com/p/engineers-are-obsessed-with-the-how)\nSep 5 • [SeattleDataGuy](https://substack.com/@seattledataguy)\n43\n6\n4\n![](../images/body-3c2dceda7ab2bc94df38.jpeg)\n### August 2026\n[From Vibe Coding to the AI Software Factory](https://seattledataguy.substack.com/p/from-vibe-coding-to-the-ai-software)\n[How I’ve Seen Engineering Teams Use LLMs Over The Past Few Years](https://seattledataguy.substack.com/p/from-vibe-coding-to-the-ai-software)\nAug 26 • [SeattleDataGuy](https://substack.com/@seattledataguy)\n32\n2\n3\n![](../images/body-d52dba8ee5c1b4485f21.jpeg)\n[The Problem With Asking Employees to “Take Ownership”](https://seattledataguy.substack.com/p/the-problem-with-asking-employees)\n[And then punishing their initiative](https://seattledataguy.substack.com/p/the-problem-with-asking-employees)\nAug 10 • [SeattleDataGuy](https://substack.com/@seattledataguy)\n61\n3\n8\n![](../images/body-f79ca990f734d84500e4.jpeg)\n### July 2026\n[Your Data Warehouse Isn’t Integrated Just Because the Tables Are in One Place](https://seattledataguy.substack.com/p/your-data-warehouse-isnt-integrated)\n[Centralization is not integration!](https://seattledataguy.substack.com/p/your-data-warehouse-isnt-integrated)\nJul 18 • [SeattleDataGuy](https://substack.com/@seattledataguy)\n61\n2\n2\n![](../images/body-48f44d911add93aa6005.png)\n[Forward Deployed Engineering Is About To Get Diluted](https://seattledataguy.substack.com/p/forward-deployed-engineering-is-about)\n[But AI vendors need enterprise adoption at any cost](https://seattledataguy.substack.com/p/forward-deployed-engineering-is-about)\nJul 6 • [SeattleDataGuy](https://substack.com/@seattledataguy)\n61\n3\n4\n![](../images/body-ea32991c76ba001042f5.png)\n### June 2026\n[In 2026 The Data Fundamentals Matter More Than Ever](https://seattledataguy.substack.com/p/in-2026-the-data-fundamentals-matter)\n[We are headed towards a massive data mess otherwise](https://seattledataguy.substack.com/p/in-2026-the-data-fundamentals-matter)\nJun 13 • [SeattleDataGuy](https://substack.com/@seattledataguy)\n82\n1\n4\n![](../images/body-c5dcb7df1fdd48f9ce0b.jpeg)\n### May 2026\n[If AI Can Replace Workers, Why Is It Hiring Consultants?](https://seattledataguy.substack.com/p/if-ai-can-replace-workers-why-is)\n[Last week, I noticed that Anthropic was hiring for a partner success manager.](https://seattledataguy.substack.com/p/if-ai-can-replace-workers-why-is)\nMay 26 • [SeattleDataGuy](https://substack.com/@seattledataguy)\n53\n4\n![](../images/body-128911e374e749231e41.png)\n[What Leading a Data Team Actually Looks Like Right Now](https://seattledataguy.substack.com/p/what-leading-a-data-team-actually)\n[Even with AI](https://seattledataguy.substack.com/p/what-leading-a-data-team-actually)\nMay 14 • [SeattleDataGuy](https://substack.com/@seattledataguy)\n61\n1\n5\n![](../images/body-8f9762559cab76fb3888.png)\n### April 2026\n[The 5 Silent Failures in Data Pipelines](https://seattledataguy.substack.com/p/the-5-silent-failures-in-data-pipelines)\n[How Your Pipelines Lie to You Without Throwing a Single Error](https://seattledataguy.substack.com/p/the-5-silent-failures-in-data-pipelines)\nApr 24 • [SeattleDataGuy](https://substack.com/@seattledataguy)\n68\n5\n4\n![](../images/body-b162f8f4529ee45aa7b1.jpeg)\n[Data Pipeline Foundations - Everything You Need To Know About Data Pipelines](https://seattledataguy.substack.com/p/data-pipeline-foundations-everything)\n[Hi, fellow future and current Data Leaders; Ben here 👋](https://seattledataguy.substack.com/p/data-pipeline-foundations-everything)\nApr 18 • [SeattleDataGuy](https://substack.com/@seattledataguy)\n107\n1\n6\n![](../images/body-915d7cfd2b127b0b2d17.webp)\n[Daily Tasks With Data Pipelines - Data Quality Checks And The Problem With Noisy Checks](https://seattledataguy.substack.com/p/daily-tasks-with-data-pipelines-data)\n[Every morning, your team wakes up to 137 data quality alerts.](https://seattledataguy.substack.com/p/daily-tasks-with-data-pipelines-data)\nApr 7 • [SeattleDataGuy](https://substack.com/@seattledataguy)\n55\n2\n4\n![](../images/body-527c574f57a117f52ab7.png)\n### March 2026\n[You Will Know Nothing And Be Happy](https://seattledataguy.substack.com/p/you-will-know-nothing-and-be-happy)\n[It’s 2030, and your boss just asked you to pull data to help better segment your users and understand their behaviors.](https://seattledataguy.substack.com/p/you-will-know-nothing-and-be-happy)\nMar 25 • [SeattleDataGuy](https://substack.com/@seattledataguy)\n77\n6\n6\n![](../images/body-b1b8807af21aa950c392.gif)\n[Full Refresh vs Incremental Pipelines](https://seattledataguy.substack.com/p/full-refresh-vs-incremental-pipelines)\n[Tradeoffs Every Data Team Should Know](https://seattledataguy.substack.com/p/full-refresh-vs-incremental-pipelines)\nMar 17 • [SeattleDataGuy](https://substack.com/@seattledataguy)\n72\n4\n6\n![](../images/body-330b3c66d917116801e4.png)\n[Layer by Layer, We Built Data Systems No One Understands](https://seattledataguy.substack.com/p/layer-by-layer-we-built-data-systems)\n[How data stacks turn into fractals](https://seattledataguy.substack.com/p/layer-by-layer-we-built-data-systems)\nMar 2 • [SeattleDataGuy](https://substack.com/@seattledataguy)\n68\n4\n4\n![](../images/body-71da2052d08b8af19cdf.png)\n### February 2026\n[Backfills - The Necessary Evil of Data Engineering](https://seattledataguy.substack.com/p/backfills-the-necessary-evil-of-data)\n[Why backfills happen, why we hate them, and how to handle them without breaking trust](https://seattledataguy.substack.com/p/backfills-the-necessary-evil-of-data)\nFeb 23 • [SeattleDataGuy](https://substack.com/@seattledataguy)\n97\n4\n9\n![](../images/body-62bd4e94c89649afe27f.png)\n[Why Data Pipelines Exist](https://seattledataguy.substack.com/p/why-data-pipelines-exist)\n[Beyond Moving Data From Point A To B](https://seattledataguy.substack.com/p/why-data-pipelines-exist)\nFeb 9 • [SeattleDataGuy](https://substack.com/@seattledataguy)\n77\n4\n9\n![](../images/body-40d062b39fe16b422b99.png)\n### January 2026\n[5 Key Predictions for the Data Industry in 2026](https://seattledataguy.substack.com/p/5-key-predictions-for-the-data-industry-b7c)\n[Hype Cycles, Rebrands, and the Messy Reality of Data](https://seattledataguy.substack.com/p/5-key-predictions-for-the-data-industry-b7c)\nJan 31 • [SeattleDataGuy](https://substack.com/@seattledataguy)\n68\n11\n4\n![](../images/body-2807214692dfd449424d.png)\n[The Analytical Skills No One Teaches You](https://seattledataguy.substack.com/p/the-analytical-skills-no-one-teaches)\n[Estimation, Baselines, Root Cause Analysis, and Metrics That Actually Matter](https://seattledataguy.substack.com/p/the-analytical-skills-no-one-teaches)\nJan 23 • [SeattleDataGuy](https://substack.com/@seattledataguy) and [Olga Berezovsky](https://substack.com/@dataanalysis)\n183\n5\n24\n![](../images/body-dda925e92b8728c9716b.png)\n[What It Actually Takes to Build a Data Pipeline System](https://seattledataguy.substack.com/p/what-it-actually-takes-to-build-a)\n[A breakdown of the components, tradeoffs, and realities of building your own data pipeline system](https://seattledataguy.substack.com/p/what-it-actually-takes-to-build-a)\nJan 14 • [SeattleDataGuy](https://substack.com/@seattledataguy)\n65\n3\n9\n![](../images/body-9aa6c9360ba03a826393.png)\n[Common Data Pipeline Patterns You’ll See in the Real World](https://seattledataguy.substack.com/p/common-data-pipeline-patterns-youll)\n[A practical look at the many ways data pipelines show up inside real companies](https://seattledataguy.substack.com/p/common-data-pipeline-patterns-youll)\nJan 5 • [SeattleDataGuy](https://substack.com/@seattledataguy)\n86\n3\n8\n![](../images/body-e5208e2a92a7e87b0edf.png)\n### December 2025\n[Snowflake vs Databricks Is the Wrong Debate](https://seattledataguy.substack.com/p/snowflake-vs-databricks-is-the-wrong)\n[Winning the Data Stack Role by Role](https://seattledataguy.substack.com/p/snowflake-vs-databricks-is-the-wrong)\nDec 12, 2025 • [SeattleDataGuy](https://substack.com/@seattledataguy)\n54\n2\n4\n![](../images/body-98de99a07dba508a595a.jpeg)\n[How I Run System Design Interviews for Data Engineers ](https://seattledataguy.substack.com/p/how-i-run-system-design-interviews)\n[Why System Design Still Matters](https://seattledataguy.substack.com/p/how-i-run-system-design-interviews)\nDec 9, 2025 • [SeattleDataGuy](https://substack.com/@seattledataguy) and [mehdio](https://substack.com/@mehdio)\n129\n4\n12\n![](../images/body-ef07b072705783a20318.png)\n[Translating Data Buzzwords Into Real Requirements](https://seattledataguy.substack.com/p/translating-data-buzzwords-into-real)\n[Bridging the Communication Gap Between Data and the Business](https://seattledataguy.substack.com/p/translating-data-buzzwords-into-real)\nDec 2, 2025 • [SeattleDataGuy](https://substack.com/@seattledataguy)\n49\n2\n8\n![](../images/body-b1c9029b7b392003b0dc.png)\n© 2026 SeattleDataGuy · [Privacy](https://substack.com/privacy) ∙ [Terms](https://substack.com/tos) ∙ [Collection notice](https://substack.com/ccpa#personal-data-collected)\n[ Start your Substack](https://substack.com/signup?utm_source=substack&utm_medium=web&utm_content=footer)[Get the app](https://substack.com/app/app-store-redirect?utm_campaign=app-marketing&utm_content=web-footer-button)\n[Substack](https://substack.com) is the home for great culture\n",
  "metadata": {
    "title": "SeattleDataGuy’s Newsletter | Substack",
    "description": "Learn About End-To-End Data Flows (Data Engineering, MLOps, and Data Science). Click to read SeattleDataGuy’s Newsletter, a Substack publication with hundreds of thousands of subscribers.",
    "keywords": null,
    "author": "SeattleDataGuy",
    "og:type": "article",
    "og:title": "SeattleDataGuy’s Newsletter | Substack",
    "og:description": "Learn About End-To-End Data Flows (Data Engineering, MLOps, and Data Science). Click to read SeattleDataGuy’s Newsletter, a Substack publication with hundreds of thousands of subscribers.",
    "og:image": "https://substackcdn.com/image/fetch/$s_!w1W4!,f_auto,q_auto:best,fl_progressive:steep/https%3A%2F%2Fseattledataguy.substack.com%2Ftwitter%2Fsubscribe-card.jpg%3Fv%3D977930225%26version%3D9",
    "og:url": "https://seattledataguy.substack.com/complete-guide-data-pipelines",
    "twitter:title": "SeattleDataGuy’s Newsletter | Substack",
    "twitter:description": "Learn About End-To-End Data Flows (Data Engineering, MLOps, and Data Science). Click to read SeattleDataGuy’s Newsletter, a Substack publication with hundreds of thousands of subscribers.",
    "twitter:image": "https://substackcdn.com/image/fetch/$s_!w1W4!,f_auto,q_auto:best,fl_progressive:steep/https%3A%2F%2Fseattledataguy.substack.com%2Ftwitter%2Fsubscribe-card.jpg%3Fv%3D977930225%26version%3D9",
    "twitter:card": "summary_large_image"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 302,
    "error": "",
    "retried": true
  },
  "code_blocks": [
    {
      "language": "txt",
      "file": "attachments/178d38f5da52/code/block_000.txt",
      "chars": 18
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!qCMC!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3b166ca1-031b-4aeb-a775-4d9a8c58482d_700x449.jpeg",
      "bytes": 18674,
      "reused": true,
      "local_path": "attachments/178d38f5da52/images/013.jpeg",
      "src": "https://substackcdn.com/image/fetch/$s_!qCMC!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3b166ca1-031b-4aeb-a775-4d9a8c58482d_700x449.jpeg",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!mTXd!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F24cbd1fd-4466-40d6-b5a7-c51034015ddc_1280x1944.jpeg",
      "bytes": 18973,
      "reused": false,
      "content_type": "image/jpeg",
      "local_path": "attachments/178d38f5da52/images/body-3c2dceda7ab2bc94df38.jpeg",
      "src": "https://substackcdn.com/image/fetch/$s_!mTXd!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F24cbd1fd-4466-40d6-b5a7-c51034015ddc_1280x1944.jpeg",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!sBjN!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe3d550a3-9600-4947-9fd4-594ef43f29e8_500x568.jpeg",
      "bytes": 14963,
      "reused": false,
      "content_type": "image/jpeg",
      "local_path": "attachments/178d38f5da52/images/body-d52dba8ee5c1b4485f21.jpeg",
      "src": "https://substackcdn.com/image/fetch/$s_!sBjN!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe3d550a3-9600-4947-9fd4-594ef43f29e8_500x568.jpeg",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!L73p!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F84c34b8b-d24b-4195-9b9f-ca0c36597b99_500x563.jpeg",
      "bytes": 13850,
      "reused": false,
      "content_type": "image/jpeg",
      "local_path": "attachments/178d38f5da52/images/body-f79ca990f734d84500e4.jpeg",
      "src": "https://substackcdn.com/image/fetch/$s_!L73p!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F84c34b8b-d24b-4195-9b9f-ca0c36597b99_500x563.jpeg",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!1cSv!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F11df16c3-b97d-44ca-bf4a-b89382126842_1024x768.png",
      "bytes": 25220,
      "reused": false,
      "content_type": "image/jpeg",
      "local_path": "attachments/178d38f5da52/images/body-48f44d911add93aa6005.png",
      "src": "https://substackcdn.com/image/fetch/$s_!1cSv!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F11df16c3-b97d-44ca-bf4a-b89382126842_1024x768.png",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!iAlK!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F963254e7-89cf-4d49-9090-d8165e6b9e19_1024x768.png",
      "bytes": 24048,
      "reused": false,
      "content_type": "image/jpeg",
      "local_path": "attachments/178d38f5da52/images/body-ea32991c76ba001042f5.png",
      "src": "https://substackcdn.com/image/fetch/$s_!iAlK!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F963254e7-89cf-4d49-9090-d8165e6b9e19_1024x768.png",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!SS1R!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F70480ca5-e6e0-47f7-a67f-8dc3bcd2047b_667x375.jpeg",
      "bytes": 14148,
      "reused": false,
      "content_type": "image/jpeg",
      "local_path": "attachments/178d38f5da52/images/body-c5dcb7df1fdd48f9ce0b.jpeg",
      "src": "https://substackcdn.com/image/fetch/$s_!SS1R!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F70480ca5-e6e0-47f7-a67f-8dc3bcd2047b_667x375.jpeg",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!Adu-!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb4afee1f-eb04-4c2b-9605-0fa564b2832a_1024x768.png",
      "bytes": 22326,
      "reused": false,
      "content_type": "image/jpeg",
      "local_path": "attachments/178d38f5da52/images/body-128911e374e749231e41.png",
      "src": "https://substackcdn.com/image/fetch/$s_!Adu-!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb4afee1f-eb04-4c2b-9605-0fa564b2832a_1024x768.png",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!E5pW!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff5d8caa8-208b-4a79-b481-db5f47a55bb8_1024x768.png",
      "bytes": 12299,
      "reused": false,
      "content_type": "image/jpeg",
      "local_path": "attachments/178d38f5da52/images/body-8f9762559cab76fb3888.png",
      "src": "https://substackcdn.com/image/fetch/$s_!E5pW!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff5d8caa8-208b-4a79-b481-db5f47a55bb8_1024x768.png",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!IRNE!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb7284dfc-0dd7-4512-b795-d1c1d7bdbbc2_500x500.jpeg",
      "bytes": 16698,
      "reused": false,
      "content_type": "image/jpeg",
      "local_path": "attachments/178d38f5da52/images/body-b162f8f4529ee45aa7b1.jpeg",
      "src": "https://substackcdn.com/image/fetch/$s_!IRNE!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb7284dfc-0dd7-4512-b795-d1c1d7bdbbc2_500x500.jpeg",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!PXKy!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9d790de7-f944-4efc-b790-a97e8893f268_1024x768.webp",
      "bytes": 29394,
      "reused": false,
      "content_type": "image/jpeg",
      "local_path": "attachments/178d38f5da52/images/body-915d7cfd2b127b0b2d17.webp",
      "src": "https://substackcdn.com/image/fetch/$s_!PXKy!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9d790de7-f944-4efc-b790-a97e8893f268_1024x768.webp",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!rMpg!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fed2ad224-d577-4df4-b5e9-7d90f4705f07_1024x768.png",
      "bytes": 30801,
      "reused": false,
      "content_type": "image/jpeg",
      "local_path": "attachments/178d38f5da52/images/body-527c574f57a117f52ab7.png",
      "src": "https://substackcdn.com/image/fetch/$s_!rMpg!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fed2ad224-d577-4df4-b5e9-7d90f4705f07_1024x768.png",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!S0v3!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F074ccb16-c874-4afe-a2a0-75e36bdf6c7b_576x315.gif",
      "bytes": 4699867,
      "reused": false,
      "content_type": "image/gif",
      "local_path": "attachments/178d38f5da52/images/body-b1b8807af21aa950c392.gif",
      "src": "https://substackcdn.com/image/fetch/$s_!S0v3!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F074ccb16-c874-4afe-a2a0-75e36bdf6c7b_576x315.gif",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!dL06!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcd842aa5-5950-4c10-bebc-ae93374176f1_1024x768.png",
      "bytes": 14530,
      "reused": false,
      "content_type": "image/jpeg",
      "local_path": "attachments/178d38f5da52/images/body-330b3c66d917116801e4.png",
      "src": "https://substackcdn.com/image/fetch/$s_!dL06!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcd842aa5-5950-4c10-bebc-ae93374176f1_1024x768.png",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!MWgB!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5dfa805d-81b2-4a61-8685-e0130eae81a9_1024x768.png",
      "bytes": 15025,
      "reused": false,
      "content_type": "image/jpeg",
      "local_path": "attachments/178d38f5da52/images/body-71da2052d08b8af19cdf.png",
      "src": "https://substackcdn.com/image/fetch/$s_!MWgB!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5dfa805d-81b2-4a61-8685-e0130eae81a9_1024x768.png",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!6Su2!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb664aa03-fe21-4c43-b5b5-dcdd154cbfb1_1024x768.png",
      "bytes": 32244,
      "reused": false,
      "content_type": "image/jpeg",
      "local_path": "attachments/178d38f5da52/images/body-62bd4e94c89649afe27f.png",
      "src": "https://substackcdn.com/image/fetch/$s_!6Su2!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb664aa03-fe21-4c43-b5b5-dcdd154cbfb1_1024x768.png",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!hIcA!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe0c8cb1c-13ce-4acc-8731-ce27d697190a_1024x768.png",
      "bytes": 24813,
      "reused": false,
      "content_type": "image/jpeg",
      "local_path": "attachments/178d38f5da52/images/body-40d062b39fe16b422b99.png",
      "src": "https://substackcdn.com/image/fetch/$s_!hIcA!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe0c8cb1c-13ce-4acc-8731-ce27d697190a_1024x768.png",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!74LM!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F17c2401d-dd70-4bdf-986d-14b0a3c7c50d_1024x768.png",
      "bytes": 25995,
      "reused": false,
      "content_type": "image/jpeg",
      "local_path": "attachments/178d38f5da52/images/body-2807214692dfd449424d.png",
      "src": "https://substackcdn.com/image/fetch/$s_!74LM!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F17c2401d-dd70-4bdf-986d-14b0a3c7c50d_1024x768.png",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!IO_a!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1fc71e6a-f351-4775-ac3d-60240e16d141_1024x768.png",
      "bytes": 16492,
      "reused": false,
      "content_type": "image/jpeg",
      "local_path": "attachments/178d38f5da52/images/body-dda925e92b8728c9716b.png",
      "src": "https://substackcdn.com/image/fetch/$s_!IO_a!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1fc71e6a-f351-4775-ac3d-60240e16d141_1024x768.png",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!FqZB!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2215cb85-d988-4445-b5df-f14a31258838_1024x768.png",
      "bytes": 11815,
      "reused": false,
      "content_type": "image/jpeg",
      "local_path": "attachments/178d38f5da52/images/body-9aa6c9360ba03a826393.png",
      "src": "https://substackcdn.com/image/fetch/$s_!FqZB!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2215cb85-d988-4445-b5df-f14a31258838_1024x768.png",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!uaVa!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8efec60d-abc1-4d62-9ce7-730de55029e0_1024x768.png",
      "bytes": 18735,
      "reused": false,
      "content_type": "image/jpeg",
      "local_path": "attachments/178d38f5da52/images/body-e5208e2a92a7e87b0edf.png",
      "src": "https://substackcdn.com/image/fetch/$s_!uaVa!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8efec60d-abc1-4d62-9ce7-730de55029e0_1024x768.png",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!rn6Q!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdd6ccf9a-698a-4253-9647-186cc436286f_1456x710.jpeg",
      "bytes": 35909,
      "reused": false,
      "content_type": "image/jpeg",
      "local_path": "attachments/178d38f5da52/images/body-98de99a07dba508a595a.jpeg",
      "src": "https://substackcdn.com/image/fetch/$s_!rn6Q!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdd6ccf9a-698a-4253-9647-186cc436286f_1456x710.jpeg",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!m3PJ!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4ebe31df-cf04-4b90-af59-1fed9b125f19_1920x1080.png",
      "bytes": 18782,
      "reused": false,
      "content_type": "image/jpeg",
      "local_path": "attachments/178d38f5da52/images/body-ef07b072705783a20318.png",
      "src": "https://substackcdn.com/image/fetch/$s_!m3PJ!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4ebe31df-cf04-4b90-af59-1fed9b125f19_1920x1080.png",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!oNid!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0e9cb499-b2ca-4b78-a454-4c28a5840351_1024x768.png",
      "bytes": 9563,
      "reused": false,
      "content_type": "image/jpeg",
      "local_path": "attachments/178d38f5da52/images/body-b1c9029b7b392003b0dc.png",
      "src": "https://substackcdn.com/image/fetch/$s_!oNid!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0e9cb499-b2ca-4b78-a454-4c28a5840351_1024x768.png",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    }
  ],
  "audio": [],
  "videos": [
    {
      "url": "https://www.youtube.com/@SeattleDataGuy/videos",
      "kind": "channel_reference",
      "status": "reference_only"
    }
  ],
  "links": [],
  "retry_at": "2026-10-03T07:56:51",
  "repair_version": 2,
  "repaired_at": "2026-10-20T17:33:58Z",
  "source_url": "https://seattledataguy.substack.com/p/complete-guide-data-pipelines",
  "delivery_status": "unverified",
  "discovery_status": "http_404",
  "missing_reason": "http_404",
  "repair_error": "",
  "actual_video_files": [],
  "expected_saved_body_images": 24,
  "image_discovery": "saved_markdown",
  "source_body_markdown": "[![SeattleDataGuy’s Newsletter](https://substackcdn.com/image/fetch/$s_!fov7!,w_40,h_40,c_fill,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2Fea9135cc-f9d6-4856-8596-2ca9a1655cb6_256x256.png)](https://seattledataguy.substack.com/)\n# [SeattleDataGuy’s Newsletter](https://seattledataguy.substack.com/)\nSubscribeSign in\n[Home](https://seattledataguy.substack.com/)\n[Notes](https://seattledataguy.substack.com/notes)\n[Videos](https://www.youtube.com/@SeattleDataGuy/videos)\n[🚀 Learning Tracks 🚀](https://seattledataguy.substack.com/p/learning-tracks-for-paid-members)\n[Webinars And Deep Dives](https://seattledataguy.substack.com/p/upcoming-data-events)\n[Archive](https://seattledataguy.substack.com/archive)\n[Leaderboard](https://seattledataguy.substack.com/leaderboard)\n[About](https://seattledataguy.substack.com/about)\n## Page not found\nLatestTopDiscussions\n[Do You Actually Need Real-Time Data?](https://seattledataguy.substack.com/p/do-you-actually-need-real-time-data)\n[Why “real-time” is often a business requirement that hasn’t been defined yet](https://seattledataguy.substack.com/p/do-you-actually-need-real-time-data)\nSep 25 • [SeattleDataGuy](https://substack.com/@seattledataguy)\n37\n8\n4\n![](https://substackcdn.com/image/fetch/$s_!qCMC!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3b166ca1-031b-4aeb-a775-4d9a8c58482d_700x449.jpeg)\n[Engineers Are Obsessed With The How And Often Forget Why?](https://seattledataguy.substack.com/p/engineers-are-obsessed-with-the-how)\n[Why data teams need to spend less time debating the stack and more time questioning the work](https://seattledataguy.substack.com/p/engineers-are-obsessed-with-the-how)\nSep 5 • [SeattleDataGuy](https://substack.com/@seattledataguy)\n43\n6\n4\n![](https://substackcdn.com/image/fetch/$s_!mTXd!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F24cbd1fd-4466-40d6-b5a7-c51034015ddc_1280x1944.jpeg)\n### August 2026\n[From Vibe Coding to the AI Software Factory](https://seattledataguy.substack.com/p/from-vibe-coding-to-the-ai-software)\n[How I’ve Seen Engineering Teams Use LLMs Over The Past Few Years](https://seattledataguy.substack.com/p/from-vibe-coding-to-the-ai-software)\nAug 26 • [SeattleDataGuy](https://substack.com/@seattledataguy)\n32\n2\n3\n![](https://substackcdn.com/image/fetch/$s_!sBjN!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe3d550a3-9600-4947-9fd4-594ef43f29e8_500x568.jpeg)\n[The Problem With Asking Employees to “Take Ownership”](https://seattledataguy.substack.com/p/the-problem-with-asking-employees)\n[And then punishing their initiative](https://seattledataguy.substack.com/p/the-problem-with-asking-employees)\nAug 10 • [SeattleDataGuy](https://substack.com/@seattledataguy)\n61\n3\n8\n![](https://substackcdn.com/image/fetch/$s_!L73p!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F84c34b8b-d24b-4195-9b9f-ca0c36597b99_500x563.jpeg)\n### July 2026\n[Your Data Warehouse Isn’t Integrated Just Because the Tables Are in One Place](https://seattledataguy.substack.com/p/your-data-warehouse-isnt-integrated)\n[Centralization is not integration!](https://seattledataguy.substack.com/p/your-data-warehouse-isnt-integrated)\nJul 18 • [SeattleDataGuy](https://substack.com/@seattledataguy)\n61\n2\n2\n![](https://substackcdn.com/image/fetch/$s_!1cSv!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F11df16c3-b97d-44ca-bf4a-b89382126842_1024x768.png)\n[Forward Deployed Engineering Is About To Get Diluted](https://seattledataguy.substack.com/p/forward-deployed-engineering-is-about)\n[But AI vendors need enterprise adoption at any cost](https://seattledataguy.substack.com/p/forward-deployed-engineering-is-about)\nJul 6 • [SeattleDataGuy](https://substack.com/@seattledataguy)\n61\n3\n4\n![](https://substackcdn.com/image/fetch/$s_!iAlK!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F963254e7-89cf-4d49-9090-d8165e6b9e19_1024x768.png)\n### June 2026\n[In 2026 The Data Fundamentals Matter More Than Ever](https://seattledataguy.substack.com/p/in-2026-the-data-fundamentals-matter)\n[We are headed towards a massive data mess otherwise](https://seattledataguy.substack.com/p/in-2026-the-data-fundamentals-matter)\nJun 13 • [SeattleDataGuy](https://substack.com/@seattledataguy)\n82\n1\n4\n![](https://substackcdn.com/image/fetch/$s_!SS1R!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F70480ca5-e6e0-47f7-a67f-8dc3bcd2047b_667x375.jpeg)\n### May 2026\n[If AI Can Replace Workers, Why Is It Hiring Consultants?](https://seattledataguy.substack.com/p/if-ai-can-replace-workers-why-is)\n[Last week, I noticed that Anthropic was hiring for a partner success manager.](https://seattledataguy.substack.com/p/if-ai-can-replace-workers-why-is)\nMay 26 • [SeattleDataGuy](https://substack.com/@seattledataguy)\n53\n4\n![](https://substackcdn.com/image/fetch/$s_!Adu-!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb4afee1f-eb04-4c2b-9605-0fa564b2832a_1024x768.png)\n[What Leading a Data Team Actually Looks Like Right Now](https://seattledataguy.substack.com/p/what-leading-a-data-team-actually)\n[Even with AI](https://seattledataguy.substack.com/p/what-leading-a-data-team-actually)\nMay 14 • [SeattleDataGuy](https://substack.com/@seattledataguy)\n61\n1\n5\n![](https://substackcdn.com/image/fetch/$s_!E5pW!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff5d8caa8-208b-4a79-b481-db5f47a55bb8_1024x768.png)\n### April 2026\n[The 5 Silent Failures in Data Pipelines](https://seattledataguy.substack.com/p/the-5-silent-failures-in-data-pipelines)\n[How Your Pipelines Lie to You Without Throwing a Single Error](https://seattledataguy.substack.com/p/the-5-silent-failures-in-data-pipelines)\nApr 24 • [SeattleDataGuy](https://substack.com/@seattledataguy)\n68\n5\n4\n![](https://substackcdn.com/image/fetch/$s_!IRNE!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb7284dfc-0dd7-4512-b795-d1c1d7bdbbc2_500x500.jpeg)\n[Data Pipeline Foundations - Everything You Need To Know About Data Pipelines](https://seattledataguy.substack.com/p/data-pipeline-foundations-everything)\n[Hi, fellow future and current Data Leaders; Ben here 👋](https://seattledataguy.substack.com/p/data-pipeline-foundations-everything)\nApr 18 • [SeattleDataGuy](https://substack.com/@seattledataguy)\n107\n1\n6\n![](https://substackcdn.com/image/fetch/$s_!PXKy!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9d790de7-f944-4efc-b790-a97e8893f268_1024x768.webp)\n[Daily Tasks With Data Pipelines - Data Quality Checks And The Problem With Noisy Checks](https://seattledataguy.substack.com/p/daily-tasks-with-data-pipelines-data)\n[Every morning, your team wakes up to 137 data quality alerts.](https://seattledataguy.substack.com/p/daily-tasks-with-data-pipelines-data)\nApr 7 • [SeattleDataGuy](https://substack.com/@seattledataguy)\n55\n2\n4\n![](https://substackcdn.com/image/fetch/$s_!rMpg!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fed2ad224-d577-4df4-b5e9-7d90f4705f07_1024x768.png)\n### March 2026\n[You Will Know Nothing And Be Happy](https://seattledataguy.substack.com/p/you-will-know-nothing-and-be-happy)\n[It’s 2030, and your boss just asked you to pull data to help better segment your users and understand their behaviors.](https://seattledataguy.substack.com/p/you-will-know-nothing-and-be-happy)\nMar 25 • [SeattleDataGuy](https://substack.com/@seattledataguy)\n77\n6\n6\n![](https://substackcdn.com/image/fetch/$s_!S0v3!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F074ccb16-c874-4afe-a2a0-75e36bdf6c7b_576x315.gif)\n[Full Refresh vs Incremental Pipelines](https://seattledataguy.substack.com/p/full-refresh-vs-incremental-pipelines)\n[Tradeoffs Every Data Team Should Know](https://seattledataguy.substack.com/p/full-refresh-vs-incremental-pipelines)\nMar 17 • [SeattleDataGuy](https://substack.com/@seattledataguy)\n72\n4\n6\n![](https://substackcdn.com/image/fetch/$s_!dL06!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcd842aa5-5950-4c10-bebc-ae93374176f1_1024x768.png)\n[Layer by Layer, We Built Data Systems No One Understands](https://seattledataguy.substack.com/p/layer-by-layer-we-built-data-systems)\n[How data stacks turn into fractals](https://seattledataguy.substack.com/p/layer-by-layer-we-built-data-systems)\nMar 2 • [SeattleDataGuy](https://substack.com/@seattledataguy)\n68\n4\n4\n![](https://substackcdn.com/image/fetch/$s_!MWgB!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5dfa805d-81b2-4a61-8685-e0130eae81a9_1024x768.png)\n### February 2026\n[Backfills - The Necessary Evil of Data Engineering](https://seattledataguy.substack.com/p/backfills-the-necessary-evil-of-data)\n[Why backfills happen, why we hate them, and how to handle them without breaking trust](https://seattledataguy.substack.com/p/backfills-the-necessary-evil-of-data)\nFeb 23 • [SeattleDataGuy](https://substack.com/@seattledataguy)\n97\n4\n9\n![](https://substackcdn.com/image/fetch/$s_!6Su2!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb664aa03-fe21-4c43-b5b5-dcdd154cbfb1_1024x768.png)\n[Why Data Pipelines Exist](https://seattledataguy.substack.com/p/why-data-pipelines-exist)\n[Beyond Moving Data From Point A To B](https://seattledataguy.substack.com/p/why-data-pipelines-exist)\nFeb 9 • [SeattleDataGuy](https://substack.com/@seattledataguy)\n77\n4\n9\n![](https://substackcdn.com/image/fetch/$s_!hIcA!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe0c8cb1c-13ce-4acc-8731-ce27d697190a_1024x768.png)\n### January 2026\n[5 Key Predictions for the Data Industry in 2026](https://seattledataguy.substack.com/p/5-key-predictions-for-the-data-industry-b7c)\n[Hype Cycles, Rebrands, and the Messy Reality of Data](https://seattledataguy.substack.com/p/5-key-predictions-for-the-data-industry-b7c)\nJan 31 • [SeattleDataGuy](https://substack.com/@seattledataguy)\n68\n11\n4\n![](https://substackcdn.com/image/fetch/$s_!74LM!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F17c2401d-dd70-4bdf-986d-14b0a3c7c50d_1024x768.png)\n[The Analytical Skills No One Teaches You](https://seattledataguy.substack.com/p/the-analytical-skills-no-one-teaches)\n[Estimation, Baselines, Root Cause Analysis, and Metrics That Actually Matter](https://seattledataguy.substack.com/p/the-analytical-skills-no-one-teaches)\nJan 23 • [SeattleDataGuy](https://substack.com/@seattledataguy) and [Olga Berezovsky](https://substack.com/@dataanalysis)\n183\n5\n24\n![](https://substackcdn.com/image/fetch/$s_!IO_a!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1fc71e6a-f351-4775-ac3d-60240e16d141_1024x768.png)\n[What It Actually Takes to Build a Data Pipeline System](https://seattledataguy.substack.com/p/what-it-actually-takes-to-build-a)\n[A breakdown of the components, tradeoffs, and realities of building your own data pipeline system](https://seattledataguy.substack.com/p/what-it-actually-takes-to-build-a)\nJan 14 • [SeattleDataGuy](https://substack.com/@seattledataguy)\n65\n3\n9\n![](https://substackcdn.com/image/fetch/$s_!FqZB!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2215cb85-d988-4445-b5df-f14a31258838_1024x768.png)\n[Common Data Pipeline Patterns You’ll See in the Real World](https://seattledataguy.substack.com/p/common-data-pipeline-patterns-youll)\n[A practical look at the many ways data pipelines show up inside real companies](https://seattledataguy.substack.com/p/common-data-pipeline-patterns-youll)\nJan 5 • [SeattleDataGuy](https://substack.com/@seattledataguy)\n86\n3\n8\n![](https://substackcdn.com/image/fetch/$s_!uaVa!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8efec60d-abc1-4d62-9ce7-730de55029e0_1024x768.png)\n### December 2025\n[Snowflake vs Databricks Is the Wrong Debate](https://seattledataguy.substack.com/p/snowflake-vs-databricks-is-the-wrong)\n[Winning the Data Stack Role by Role](https://seattledataguy.substack.com/p/snowflake-vs-databricks-is-the-wrong)\nDec 12, 2025 • [SeattleDataGuy](https://substack.com/@seattledataguy)\n54\n2\n4\n![](https://substackcdn.com/image/fetch/$s_!rn6Q!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdd6ccf9a-698a-4253-9647-186cc436286f_1456x710.jpeg)\n[How I Run System Design Interviews for Data Engineers ](https://seattledataguy.substack.com/p/how-i-run-system-design-interviews)\n[Why System Design Still Matters](https://seattledataguy.substack.com/p/how-i-run-system-design-interviews)\nDec 9, 2025 • [SeattleDataGuy](https://substack.com/@seattledataguy) and [mehdio](https://substack.com/@mehdio)\n129\n4\n12\n![](https://substackcdn.com/image/fetch/$s_!m3PJ!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4ebe31df-cf04-4b90-af59-1fed9b125f19_1920x1080.png)\n[Translating Data Buzzwords Into Real Requirements](https://seattledataguy.substack.com/p/translating-data-buzzwords-into-real)\n[Bridging the Communication Gap Between Data and the Business](https://seattledataguy.substack.com/p/translating-data-buzzwords-into-real)\nDec 2, 2025 • [SeattleDataGuy](https://substack.com/@seattledataguy)\n49\n2\n8\n![](https://substackcdn.com/image/fetch/$s_!oNid!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0e9cb499-b2ca-4b78-a454-4c28a5840351_1024x768.png)\n© 2026 SeattleDataGuy · [Privacy](https://substack.com/privacy) ∙ [Terms](https://substack.com/tos) ∙ [Collection notice](https://substack.com/ccpa#personal-data-collected)\n[ Start your Substack](https://substack.com/signup?utm_source=substack&utm_medium=web&utm_content=footer)[Get the app](https://substack.com/app/app-store-redirect?utm_campaign=app-marketing&utm_content=web-footer-button)\n[Substack](https://substack.com) is the home for great culture\n",
  "body_base_path": "attachments/178d38f5da52/pages/",
  "body_markdown_path": "attachments/178d38f5da52/pages/archived-body.md",
  "local_html": "attachments/178d38f5da52/pages/archived-body.html",
  "body_source": "prior_capture"
}