{
  "source_file": "0930",
  "source_platform": "Ghost",
  "author": "Avi Chawla / Daily Dose of Data Science",
  "title_listed": "Building A Context Engineering Workflow",
  "keywords": "",
  "domain": "文本处理",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://www.dailydoseofds.com/p/building-a-context-engineering-workflow/",
  "publish_date": "2026-05-31",
  "corpus_stage": "",
  "notes": "",
  "id": "d5fe63f6a38d",
  "collected_at": "2026-10-02T13:47:53",
  "title": "​Building a Context Engineering Workflow​",
  "body_markdown": "Sep 13, 2025\nMachine Learning\n​Building a Context Engineering Workflow​\n..explained step-by-step with code.\nAvi Chawla\n👉\nA few days back, we discussed covering a demo on context engineering.\nSo today, we'll build a multi-agent research assistant using context engineering principles.\nThis Agent will gather its context across 4 sources: Documents, Memory, Web search, and Arxiv.\nHere’s our workflow:\nUser submits query.\nFetch context from docs, web, arxiv API, and memory.\nPass the aggregated context to an agent for filtering.\nPass the filtered context to another agent to generate a response.\nSave the final response to memory.\nTech stack:\n​\nTensorlake\n​\nto get RAG-ready data from complex docs\n​\nZep\n​\nfor memory\n​\nFirecrawl\n​\nfor web search\n​\nMilvus\n​\nfor vector DB\nCrewAI for orchestration\nThe code is linked later in the issue.\nLet's go!\nFirst, what is context engineering (CE)?\nLLMs aren’t mind readers. They can only work with what you give them.\nPrompt engineering primarily focuses on “magic words” with an expectation of getting a better response.\nCE involves creating dynamic systems that offer:\nThe right info\nThe right tools\nIn the right format\nThis ensures the LLM can effectively complete the task.\nCrew flow\nWe'll follow a top-down approach to understand the code.\nHere's an outline of what our flow looks like:\nNote that this is one of many blueprints to implement a context engineering workflow. Your pipeline will likely vary based on the use case.\nPrepare data for RAG\nWe use Tensorlake to convert the document into RAG-ready markdown chunks for each section.\nThe extracted data can be directly embedded and stored in a vector DB without further processing.\nIndexing and retrieval\nNow that we have RAG-ready chunks along with the metadata, it's time to store them in a self-hosted Milvus vector database.\nWe retrieve the top-k most similar chunks to our query:\nBuild memory layer\nZep acts as the core memory layer of our workflow. It creates temporal knowledge graphs to organize and retrieve context for each interaction.\nWe use it to store and retrieve context from chat history and user data.\nFirecrawl web search\nWe use Firecrawl web search to fetch the latest news and developments related to the user query.\nFirecrawl's v2 endpoint provides 10x faster scraping, semantic crawling, and image search, turning any website into LLM-ready data.\nArXiv API search\nTo further support research queries, we use the arXiv API to retrieve relevant results from their data repository based on the user query.\nFilter context\nNow, we pass our combined context to the context evaluation agent that filters out irrelevant context.\nThis filtered context is then passed to the synthesizer agent that generates the final response.\nKick off the workflow\nFinally, we kick off our context engineering workflow with a query.\nBased on the query, we notice that the RAG tool, powered by Tensorlake, was the most relevant source for the LLM to generate a response.\nWe also translated this workflow into a streamlit app that:\n0:00\n/\n0:31\n1×\nProvides citations with links and metadata.\nProvides insights into relevant sources.\nThat said, the workflow explained in the thread is one of the many blueprints. Your implementation can vary.\nIn the project, we used:\nTensorlake:\nIt lets you transform any unstructured doc into AI-ready data.\n​\nGitHub Repo →\n​\nZep:\nIt lets you build human-like memory for your Agents.\n​\nGitHub Repo →\n​\nFirecrawl:\nIt lets you power LLM apps with clean data from the web.\n​\nGitHub Repo →\n​\nMilvus:\nIt gives a high-performance vector DB for scalable vector search.\n​\nGitHub Repo →\n​\n​\nYou can find the code here →\n​\nThanks for reading!\nPublished on Sep 13, 2025\nComments\nShare\nCopy link\nShare to X\nShare to Facebook\nShare to Linkedin\nCopied\nPrevious\nData and Pipeline Engineering: Distributed Processing and Workflow Orchestration\nNext\nModel Development and Optimization: Fundamentals of Development and Hyperparameter Tuning",
  "metadata": {
    "title": "​Building a Context Engineering Workflow​",
    "description": "..explained step-by-step with code.",
    "keywords": null,
    "author": null,
    "og:site_name": "Daily Dose of Data Science",
    "og:type": "article",
    "og:title": "​Building a Context Engineering Workflow​",
    "og:description": "..explained step-by-step with code.",
    "og:url": "https://www.dailydoseofds.com/p/building-a-context-engineering-workflow/",
    "og:image": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2025/10/context-engg-workflow.jpg",
    "og:image:width": "1162",
    "og:image:height": "936",
    "twitter:card": "summary_large_image",
    "twitter:title": "​Building a Context Engineering Workflow​",
    "twitter:description": "..explained step-by-step with code.",
    "twitter:url": "https://www.dailydoseofds.com/p/building-a-context-engineering-workflow/",
    "twitter:image": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2025/10/context-engg-workflow.jpg",
    "twitter:label1": "Written by",
    "twitter:data1": "Avi Chawla",
    "twitter:label2": "Filed under",
    "twitter:data2": "Machine Learning",
    "twitter:site": "@_avichawla",
    "article:published_time": "2025-09-13T09:54:00.000Z",
    "article:modified_time": "2026-05-31T11:36:27.000Z",
    "article:tag": "Machine Learning"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [],
  "images": [
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/size/w400/2026/03/avi-google.jpg",
      "bytes": 11344,
      "reused": true,
      "local_path": "attachments/d5fe63f6a38d/images/000.jpg",
      "src": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/size/w400/2026/03/avi-google.jpg",
      "alt": "Avi Chawla",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2026/01/https-3a-2f-2fsubstack-post-media-s3-amazonaws-com-2fpublic-2fimages-2f31ceb515-23f7-4b7f-b4a2-b92f08675bd2_1162x936.png",
      "bytes": 326939,
      "reused": true,
      "local_path": "attachments/d5fe63f6a38d/images/001.png",
      "src": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2026/01/https-3a-2f-2fsubstack-post-media-s3-amazonaws-com-2fpublic-2fimages-2f31ceb515-23f7-4b7f-b4a2-b92f08675bd2_1162x936.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!-ye8!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F76d7884b-9b0f-4da8-b8c7-e7f0fcd36acc_1714x1536.gif",
      "bytes": 1707374,
      "reused": true,
      "local_path": "attachments/d5fe63f6a38d/images/004.gif",
      "src": "https://substackcdn.com/image/fetch/$s_!-ye8!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F76d7884b-9b0f-4da8-b8c7-e7f0fcd36acc_1714x1536.gif",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2026/01/https-3a-2f-2fsubstack-post-media-s3-amazonaws-com-2fpublic-2fimages-2fef48db7a-8882-4a8c-ad93-f0618f66d16b_680x640.png",
      "bytes": 124881,
      "reused": true,
      "local_path": "attachments/d5fe63f6a38d/images/005.png",
      "src": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2026/01/https-3a-2f-2fsubstack-post-media-s3-amazonaws-com-2fpublic-2fimages-2fef48db7a-8882-4a8c-ad93-f0618f66d16b_680x640.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2026/01/https-3a-2f-2fsubstack-post-media-s3-amazonaws-com-2fpublic-2fimages-2fbe4038f6-30b8-455a-93f7-58a4d5ef206e_680x664.png",
      "bytes": 139834,
      "reused": true,
      "local_path": "attachments/d5fe63f6a38d/images/007.png",
      "src": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2026/01/https-3a-2f-2fsubstack-post-media-s3-amazonaws-com-2fpublic-2fimages-2fbe4038f6-30b8-455a-93f7-58a4d5ef206e_680x664.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2026/01/https-3a-2f-2fsubstack-post-media-s3-amazonaws-com-2fpublic-2fimages-2f175cd496-0d31-46c2-a25f-e4afcd2900b1_680x573.png",
      "bytes": 91133,
      "reused": true,
      "local_path": "attachments/d5fe63f6a38d/images/009.png",
      "src": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2026/01/https-3a-2f-2fsubstack-post-media-s3-amazonaws-com-2fpublic-2fimages-2f175cd496-0d31-46c2-a25f-e4afcd2900b1_680x573.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2026/01/https-3a-2f-2fsubstack-post-media-s3-amazonaws-com-2fpublic-2fimages-2f47f46c87-d449-4c06-826e-9ffe3c4248be_680x622.png",
      "bytes": 134291,
      "reused": true,
      "local_path": "attachments/d5fe63f6a38d/images/011.png",
      "src": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2026/01/https-3a-2f-2fsubstack-post-media-s3-amazonaws-com-2fpublic-2fimages-2f47f46c87-d449-4c06-826e-9ffe3c4248be_680x622.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2026/01/https-3a-2f-2fsubstack-post-media-s3-amazonaws-com-2fpublic-2fimages-2fe938b5c3-adf9-4be8-a192-c6940161eaee_680x610.png",
      "bytes": 111532,
      "reused": true,
      "local_path": "attachments/d5fe63f6a38d/images/013.png",
      "src": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2026/01/https-3a-2f-2fsubstack-post-media-s3-amazonaws-com-2fpublic-2fimages-2fe938b5c3-adf9-4be8-a192-c6940161eaee_680x610.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2026/01/https-3a-2f-2fsubstack-post-media-s3-amazonaws-com-2fpublic-2fimages-2f7eceaf3a-9a2e-48fb-9a71-357093ede756_680x594.png",
      "bytes": 95337,
      "reused": true,
      "local_path": "attachments/d5fe63f6a38d/images/015.png",
      "src": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2026/01/https-3a-2f-2fsubstack-post-media-s3-amazonaws-com-2fpublic-2fimages-2f7eceaf3a-9a2e-48fb-9a71-357093ede756_680x594.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2026/01/https-3a-2f-2fsubstack-post-media-s3-amazonaws-com-2fpublic-2fimages-2fd72f39d9-9163-4659-92b0-0b098371c2e8_680x640.png",
      "bytes": 137233,
      "reused": true,
      "local_path": "attachments/d5fe63f6a38d/images/017.png",
      "src": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2026/01/https-3a-2f-2fsubstack-post-media-s3-amazonaws-com-2fpublic-2fimages-2fd72f39d9-9163-4659-92b0-0b098371c2e8_680x640.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2026/01/https-3a-2f-2fsubstack-post-media-s3-amazonaws-com-2fpublic-2fimages-2f8a2f527f-ec0d-4f7c-b0ad-e2011db0f59a_680x607.png",
      "bytes": 98206,
      "reused": true,
      "local_path": "attachments/d5fe63f6a38d/images/019.png",
      "src": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2026/01/https-3a-2f-2fsubstack-post-media-s3-amazonaws-com-2fpublic-2fimages-2f8a2f527f-ec0d-4f7c-b0ad-e2011db0f59a_680x607.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2026/01/https-3a-2f-2fsubstack-post-media-s3-amazonaws-com-2fpublic-2fimages-2f4749419b-f034-4e85-8441-5c12837bbb91_1162x936.png",
      "bytes": 326939,
      "reused": true,
      "local_path": "attachments/d5fe63f6a38d/images/021.png",
      "src": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2026/01/https-3a-2f-2fsubstack-post-media-s3-amazonaws-com-2fpublic-2fimages-2f4749419b-f034-4e85-8441-5c12837bbb91_1162x936.png",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [
    {
      "url": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/media/2025/10/context-engineering-workflow.mp4",
      "src": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/media/2025/10/context-engineering-workflow.mp4",
      "status": "ok",
      "kind": "direct_video",
      "bytes": 14220287,
      "reused": false,
      "content_type": "application/octet-stream",
      "valid": true,
      "height": 1080,
      "width": 1588,
      "duration": "31.033333",
      "local_path": "attachments/d5fe63f6a38d/videos/repair-a9371f44d2f8e59c762f.mp4"
    }
  ],
  "links": [
    {
      "href": "https://github.com/tensorlakeai/tensorlake?ref=dailydoseofds.com",
      "label": "​ Tensorlake ​",
      "type": "a"
    },
    {
      "href": "https://github.com/getzep/graphiti?ref=dailydoseofds.com",
      "label": "​ Zep ​",
      "type": "a"
    },
    {
      "href": "https://github.com/mendableai/firecrawl?ref=dailydoseofds.com",
      "label": "​ Firecrawl ​",
      "type": "a"
    },
    {
      "href": "https://github.com/milvus-io/milvus?ref=dailydoseofds.com",
      "label": "​ Milvus ​",
      "type": "a"
    },
    {
      "href": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/media/2025/10/context-engineering-workflow.mp4",
      "label": "",
      "type": "video"
    },
    {
      "href": "https://github.com/firecrawl/firecrawl?ref=dailydoseofds.com",
      "label": "​ GitHub Repo → ​",
      "type": "a"
    },
    {
      "href": "https://github.com/patchy631/ai-engineering-hub/tree/main/context-engineering-workflow?ref=dailydoseofds.com",
      "label": "​ You can find the code here → ​",
      "type": "a"
    },
    {
      "href": "https://x.com/intent/tweet?url=https%3A%2F%2Fwww.dailydoseofds.com%2Fp%2Fbuilding-a-context-engineering-workflow%2F&text=%E2%80%8BBuilding%20a%20Context%20Engineering%20Workflow%E2%80%8B",
      "label": "Share to X",
      "type": "a"
    },
    {
      "href": "https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fwww.dailydoseofds.com%2Fp%2Fbuilding-a-context-engineering-workflow%2F",
      "label": "Share to Facebook",
      "type": "a"
    },
    {
      "href": "https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fwww.dailydoseofds.com%2Fp%2Fbuilding-a-context-engineering-workflow%2F",
      "label": "Share to Linkedin",
      "type": "a"
    },
    {
      "href": "https://www.dailydoseofds.com/mlops-crash-course-part-7/",
      "label": "Previous Data and Pipeline Engineering: Distributed Processing and Workflow Orchestration",
      "type": "a"
    },
    {
      "href": "https://www.dailydoseofds.com/mlops-crash-course-part-8/",
      "label": "Next Model Development and Optimization: Fundamentals of Development and Hyperparameter Tuning",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:47:59Z",
  "source_url": "https://www.dailydoseofds.com/p/building-a-context-engineering-workflow/",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 3948,
  "expected_body_images": 12,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/d5fe63f6a38d/pages/source.html",
  "local_html": "attachments/d5fe63f6a38d/pages/article.html",
  "body_base_path": "attachments/d5fe63f6a38d/pages/",
  "body_markdown_path": "attachments/d5fe63f6a38d/pages/article.md"
}