{
  "source_file": "0930",
  "source_platform": "Hashnode",
  "author": "Jaival Suthar",
  "title_listed": "Scraper-CI: Building a Web Data Reliability & Crawl-Governance Control Plane With Bright Data",
  "keywords": "",
  "domain": "机器学习",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://jaivalsuthar.hashnode.dev/scraper-ci-web-data-reliability-control-plane",
  "publish_date": "2026-08-23",
  "corpus_stage": "",
  "notes": "",
  "id": "4a51670fa0c2",
  "collected_at": "2026-10-02T14:12:15",
  "title": "Scraper-CI: Web Data Reliability with Bright Data",
  "body_markdown": "Scraper-CI: Building a Web Data Reliability & Crawl-Governance Control Plane With Bright Data\nA technical deep dive into source profiling, bounded crawl policies, acquisition routing, data reliability, diagnosis, self-healing, verification, and downstream intelligence.\nUpdated\nAugust 23, 2026\n•\n17\nmin read\n•\nView as Markdown\nJ\nJaival Suthar\nI write technical articles that analyze real world engineering problems through the lens of production systems.\nMy work focuses on architectural tradeoffs, UI layer system design, performance optimization, and reliability patterns derived from shipping and maintaining scalable applications.\nThe problem: a successful scraper can still be wrong\nA scraper returning HTTP 200 is not necessarily healthy. It can return thousands of records, finish without an exception, and still be producing bad data — a website changes its structure, a field disappears, pagination behaves differently, a page starts returning a different template, a required field goes empty. The scraper keeps running the whole time.\nFrom the infrastructure's perspective, everything looks fine:\nHTTP 200\n   ↓\nRecords returned\n   ↓\nJob completed\nFrom the application's perspective:\nThe data is wrong.\nThat gap is what I wanted to address with\nScraper-CI\n: a web-data reliability and recovery control plane with Bright Data at its center. The goal was not to build another scraper. The goal was to build the layer around the scraper that can answer:\nWhat does this source look like?\nWhat should actually be crawled?\nWhat should not be crawled?\nWhich acquisition capability makes sense?\nDid the extraction still satisfy its contract?\nIf the data degraded, why?\nCan the system recover?\nDid the recovery actually fix the problem?\nCan downstream applications consume the resulting data without knowing how it was acquired?\nThe resulting lifecycle is:\nPROFILE\n   ↓\nPOLICY\n   ↓\nROUTE\n   ↓\nACQUIRE\n   ↓\nVALIDATE\n   ↓\nDIAGNOSE\n   ↓\nHEAL\n   ↓\nVERIFY\n   ↓\nCONSUME\nThis article explains how I built that system, why I designed the boundaries this way, and why treating \"a scraper ran successfully\" and \"the scraper's data is trustworthy\" as two separate questions turned out to be the central design decision. Consider a scraper that historically produces 100 records at 98% required-field completeness — healthy by any reasonable measure. Then the target website changes its layout, and the same scraper still returns 100 records, still returns HTTP 200, but now sits at 42% completeness. Nothing crashed. The record count didn't even move. But the dataset is clearly degraded, and no infrastructure-level signal would have caught it.\n🔗 Scraper-CI GitHub Repository:\n(\nhttps://github.com/Jaival-Suthar/sci\n)\nThat led to the central design principle behind Scraper-CI:\nA scraper should not be considered reliable merely because it runs. It should be considered reliable when the data it produces continues to satisfy the contract it was built to provide.\nScraper-CI therefore treats acquisition success and data correctness as separate concerns:\nAcquisition succeeded\n        ≠\nExtraction is trustworthy\nThat distinction drives the rest of the architecture.\nBright Data is the acquisition engine at the heart of the system\nThe first architectural decision was to\nnot rebuild the acquisition layer\n. Bright Data already provides sophisticated web-acquisition infrastructure — Scraper Studio handles navigation, pagination, sitemap loading, browser/code execution, and structured extraction, along with its own self-healing capabilities. That's valuable, mature infrastructure, and Scraper-CI is built to sit directly on top of it rather than compete with it. Bright Data isn't a bolted-on dependency here — it's the acquisition core the rest of the system is designed around.\nThe boundary is deliberate:\nBright Data\n    = acquisition (the core)\n\nScraper-CI\n    = intelligence + control + reliability + recovery\n\nConsumer\n    = domain-specific interpretation\nThis separation became the foundation for everything else.\nSource Intelligence: Profiling, Policy, and Scope\nProfile the source before deciding how to crawl it\nA website isn't simply a graph of URLs and links pointing to more URLs and links — it carries signals that help determine how it should be approached in the first place. Scraper-CI introduced a\nsource profiling layer\nfor exactly this reason. Given a target URL, the profiler inspects signals such as page type, platform, JavaScript dependency, internal and external links, canonical relationships, pagination, sitemap availability, repeated-content patterns, and domain/page relationships.\nThe profile isn't a report for its own sake — it becomes input to the next question:\nwhat should this acquisition be allowed to do?\nCrawl policy: turning source understanding into boundaries\nA scraper can have powerful navigation primitives and still lack an explicit statement of intent. I wanted Scraper-CI to make crawl scope visible and bounded, so the source profile feeds into an inferred\ncrawl policy\n:\nInternal links       ✓\nCanonical links      ✓\nPagination           ✓\nSitemap              ✓\nUnrelated external   ✕\nThe policy also expresses bounded constraints — depth ≤ 2, pages ≤ 50, no external domains, that kind of thing. The important distinction is that the crawler\nhas\ncapabilities, while the policy defines the intended boundary of those capabilities. That's why I think of this as\ncrawl governance\nrather than just another crawler implementation.\nSitemap support: primitive versus abstraction\nThere's an important distinction worth being precise about here. Bright Data already supports sitemap loading in Scraper Studio, so I wouldn't claim Bright Data lacks sitemap support — that would simply be wrong. Bright Data provides the primitive. Scraper-CI adds a different abstraction around it: sitemap discovery and sitemap-derived crawl scope become part of source intelligence and crawl-policy inference.\nBright Data\n    ↓\nSitemap-loading capability\n\nScraper-CI\n    ↓\nDetect sitemap-related source signals\n    ↓\nUse them as crawl intelligence\n    ↓\nBuild bounded crawl intent\n    ↓\nPass that intent into acquisition\nThe contribution isn't inventing sitemap support — it's the\ncontrol-plane abstraction around the primitive\n.\nThe policy participates in execution\nThe inferred crawl policy isn't merely stored as metadata — it influences the acquisition request itself.\nUser extraction intent\n          +\nSource-derived crawl intelligence\n          ↓\nAcquisition description\n          ↓\nBright Data collector\nThe acquisition intent can carry constraints like depth ≤ 2, pages ≤ 50, internal = 1, canonical = 1, pagination = 1, sitemap = 1, external = 0. This creates two layers of protection: first, the acquisition system receives the intended crawl constraints up front (\nacquisition guidance\n); second, Scraper-CI independently evaluates the resulting acquisition against the expected state afterward (\nindependent validation\n). The policy isn't just documentation — it participates in execution and is subsequently checked.\nAcquisition Routing\nFrom profiling to acquisition routing\nOnce the source has been profiled and the crawl policy inferred, the next question is which acquisition capability should handle this source. This became the\nCapability Router\n. Instead of assuming every URL gets the same treatment, the system evaluates source signals and produces an acquisition decision.\nSource Profile\n      │\n      ▼\nCapability Router\n      │\n      ├── primary capability\n      ├── fallback capabilities\n      ├── evidence\n      └── confidence\nA routing decision might look conceptually like: source signals → recommended capability → confidence: 78% → acquisition. That confidence score is deliberately not presented as a guarantee — it means the router found stronger evidence for this strategy, not that the scraper is guaranteed to succeed. Reliability is measured from actual extraction results, not from routing confidence.\nCapability abstraction\nScraper-CI uses an abstraction around acquisition capabilities rather than hard-coding the rest of the control plane to one implementation. The current capability model includes pipeline, unlocker, browser, scraper_studio, and search. Some are concrete execution paths, others are extensible capability contracts — that distinction is intentional, since I didn't want the architecture to pretend an executor exists when it doesn't.\nCapability contract\n        ↓\nConcrete adapter\n        ↓\nAcquisition provider\nThis means additional acquisition mechanisms can be introduced later without rewriting source profiling, crawl policy, reliability, diagnosis, recovery, verification, or downstream consumers. The result:\nacquisition can evolve without becoming the architecture.\nReliability, Diagnosis, and Recovery\nValidate the data after acquisition\nOnce Bright Data returns structured data, Scraper-CI asks a different question: is the result still trustworthy? A simplified pipeline runs schema validation, required-field validation, completeness, uniqueness, record-count drift, record comparison, historical drift, and health scoring — landing on HEALTHY, DEGRADED, or INVALID. The system never treats \"records exist\" as proof of correctness; it evaluates the records themselves against schema validity, required-field completeness, uniqueness, record-count drift, record-level differences, and historical reliability. The goal throughout is detecting\nsilent degradation\n— the kind of failure that never throws an exception.\nDiagnosis: turning failure into an explanation\nA health score alone isn't enough. If the system reports \"Health: 54, Status: DEGRADED,\" the next question is why. Scraper-CI's diagnosis layer classifies problems: missing required fields point to extraction degradation, schema mismatch points to contract violation, blocked acquisition points to an acquisition problem, high JavaScript dependency points to an execution strategy problem. This creates a clean separation — validation answers\nwhat\nis wrong, diagnosis answers\nwhy\nit's wrong, and the recovery planner answers\nwhat to try next\n. Diagnosis is persisted with the run and becomes input to recovery planning.\nDiagnosis-driven recovery\nA common automation pattern is to retry exactly the same thing after a failure, but that's not always useful. If the problem is a missing field, repeating the same acquisition may just reproduce the same missing field. If acquisition is blocked, the system may need a different capability entirely. If the page is JavaScript-heavy, the execution strategy needs to change. Scraper-CI makes recovery diagnosis-driven:\nDiagnosis\n    │\n    ├── Missing field       → Self-healing\n    ├── Schema failure       → Self-healing\n    ├── Blocked acquisition  → Unlocker strategy\n    ├── JS dependency        → Browser strategy\n    └── Unknown              → Fallback strategy\nThe recovery planner is deterministic and independently testable.\nAutomatic healing is not the end of the workflow\nWhen an acquisition finishes in a degraded or invalid state, Scraper-CI can enter an automatic recovery path. The current workflow allows up to two automatic healing attempts per incident:\nRun → DEGRADED/INVALID → Healing attempt 1 → Re-run → Verify\n   → still degraded? → Healing attempt 2 → Re-run → Verify\nThe important part is what happens\nafter\nhealing. A successful repair operation doesn't automatically mean the data is correct — Scraper-CI runs the extraction again and independently evaluates the result. The loop is observe → diagnose → heal → re-run → measure.\nConstrained recovery\nI didn't want recovery to become \"something is broken, fix it.\" Instead, Scraper-CI generates a diagnosis-derived repair instruction, carrying target information, observed degradation reasons, current field completeness, the extraction contract, preservation rules, and scope constraints. The repair instructions emphasize preserving existing fields and meanings, not expanding crawl scope, not introducing unrelated navigation, and making the smallest extraction change necessary. The goal is to restore the extraction contract with the smallest reasonable change — a repair that fixes one field by dramatically changing what the scraper crawls isn't actually a good repair.\nAutomatic healing versus manual correction\nThere's an important distinction between failure recovery and intentional contract evolution.\nAutomatic healing\napplies when something degraded and the system should restore the existing extraction contract.\nManual correction\napplies when the extraction still works but the desired contract has intentionally changed — for example, adding an author field while preserving the existing extraction. That's not a failure; it's an intentional requirement change. Keeping these separate prevents every change in requirements from being treated as scraper failure.\nVerification is a separate phase\nOne of the easiest mistakes in a self-healing system is treating the repair API response as proof that the repair worked. I deliberately avoided that — recovery is incomplete until the resulting extraction has been independently evaluated.\nExisting state → Repair → New acquisition → Independent validation\n    → Before/after comparison → Verification result\nVerification evaluates schema pass, completeness pass, health improvement, regression state, record-level differences, recovery delta, and diagnosis state, landing on either REPAIR VERIFIED or REPAIR REJECTED. This distinction is fundamental:\na repair action is not the same thing as a repaired dataset.\nThe Operator Control Plane\nA UI over the same domain logic\nAll of this eventually needed a UI. The Scraper-CI dashboard is an operator control plane over the backend orchestration, with main actions covering Automated, Inspect, Profile, Status, Reliability, Diagnose, Correct data, Verify, and Open URL. These aren't independent frontend implementations — the frontend calls the backend API, which invokes the same domain and orchestration logic used by the CLI. The UI is a control surface, not the system itself.\nInspect and Profile answer different questions\nInspect\nanswers which acquisition capability to use — it exposes source intelligence, recommended capability, confidence, supporting evidence, and fallback capabilities. This is the routing view.\nProfile\nanswers what the source looks like and what crawl boundaries can be inferred — it exposes source signals, relationships, sitemap information, pagination, internal/external structure, and the inferred crawl policy. This is the source-intelligence view. Keeping these separate makes the architecture understandable to an operator.\nReliability history\nA single run tells you whether something worked today. It doesn't tell you whether the scraper is reliable. Scraper-CI therefore persists historical reliability state — run health, health score, record count, validation state, degradation, recovery attempts, verification state — turning reliability into a historical property rather than a single status field. That lets \"did the latest run pass?\" and \"has this target been stable over time?\" be answered separately.\nFlight Intelligence: A Real Downstream Consumer\nThe architecture\nTo prove Scraper-CI wasn't just an abstract control plane, I built a concrete downstream application:\nFlight Intelligence\n.\nPublic aviation source → Bright Data → Scraper-CI → Validation\n    → Trusted structured data → Flight Intelligence\nFlight Intelligence doesn't need to understand the acquisition mechanics — how Bright Data collected the records, which capability was selected, how the collector was created, how crawl policy was inferred, how diagnosis works, or how healing works. It just consumes structured data. That separation was intentional.\nA consumer, not part of the control plane\nFlight Intelligence is domain-specific — it understands aircraft. Scraper-CI doesn't, and shouldn't.\nScraper-CI\n    │ structured snapshot\n    ▼\nFlight Intelligence\n    ├── aircraft counts\n    ├── altitude intelligence\n    ├── speed intelligence\n    ├── aircraft identity\n    └── historical activity\nThe consumer derives observations like aircraft count, airborne versus ground activity, average and maximum altitude and speed, altitude bands, high-altitude traffic, new and disappeared aircraft, cumulative distinct aircraft, and historical activity — none of which needs to leak back into Scraper-CI. That's what makes the consumer replaceable.\nWhy this architecture matters\nThe larger idea isn't \"I built a flight dashboard\" — it's that I built a reusable reliability layer that can feed different applications. Today it's Scraper-CI feeding Flight Intelligence. Tomorrow it could just as easily feed monitoring, analytics, research, alerting, RAG, agent workflows, or any other domain-specific application, without rebuilding the acquisition and reliability infrastructure each time.\nProduction and Testing\nProduction architecture\nThe deployed system is split into independent services:\nThe backend owns the Bright Data credential; the frontend never receives the API key. Persistent control-plane state lives in PostgreSQL. Flight Intelligence's production collection is scheduled separately through GitHub Actions, running every two hours (\n0 */2 * * *\n) — so the consumer isn't backed by a static demo dataset, but by a live, recurring pipeline: scheduled workflow → scrape-ci run → Bright Data → Scraper-CI → PostgreSQL → Flight Intelligence.\nReproducibility and deterministic testing\nWeb acquisition has a hard testing problem: the web itself is nondeterministic. A live website can change between two test runs, making it difficult to tell whether a failure came from the system or from the external source changing underneath it. Scraper-CI includes deterministic tests and benchmark scenarios covering acquisition behavior, API contracts, health scoring, completeness degradation, schema validation, capability routing, crawl policy inference, relationship classification, diagnosis, drift, recovery planning, recovery guards, reliability history, and repair verification — the goal being to keep the intelligence and reliability layers reproducible even when the external web isn't.\nLessons and Honest Boundaries\nWhat I learned\n1) HTTP success is a weak signal.\nAn HTTP response tells you a request succeeded, not that the extracted dataset is correct — and that single fact changes the architecture considerably.\n2) Crawl scope should be explicit.\nOnce a system starts following links, \"what does this scraper crawl?\" becomes an architectural question. Encoding scope as a policy makes that intent visible, testable, and enforceable.\n3) Don't confuse primitives with policy.\nBright Data already provides powerful crawling and acquisition primitives. The interesting problem is deciding which primitives to use, how to bound them, what evidence supports the decision, and whether the resulting acquisition satisfied the intended contract.\n4) Recovery without verification is dangerous.\nA healing operation can report success while the resulting data remains wrong. Heal ≠ verify — recovery must end with independent verification.\n5) Keep domain intelligence downstream.\nAircraft-specific logic doesn't belong in a generic scraping reliability layer, and neither does ecommerce, jobs, news, or finance logic. Scraper-CI produces trusted structured snapshots; consumers interpret them. That boundary is what makes the system reusable.\nWhat Scraper-CI does not claim\nWorth stating explicitly: Scraper-CI does\nnot\nclaim to have invented web scraping, sitemap parsing, pagination, browser automation, anti-bot infrastructure, proxy infrastructure, scraper self-healing, or structured extraction. Bright Data already provides significant, mature infrastructure in all of these areas — it's the acquisition core this entire system is built to sit on top of. The architectural contribution is the layer around it: source profiling, crawl policy, acquisition routing, reliability evaluation, diagnosis, recovery planning, and independent verification. The project is stronger for being honest about that boundary.\nPutting It All Together\nThe complete architecture\nThis is the system I ended up building.\nThe bigger idea\nThe web is not a static API. Selectors change, schemas change, page relationships change, navigation changes, content changes. And the most dangerous failures are often not crashes — they're successful executions that produce increasingly incorrect data. That's why web acquisition needs a control plane that treats reliability as a lifecycle rather than a boolean: understand → bound → route → acquire → validate → diagnose → recover → verify → consume.\nBright Data is the acquisition core at the heart of this system. Scraper-CI provides the intelligence, control, reliability, and recovery around it. Flight Intelligence demonstrates what can be built on top. That separation is ultimately the point.\nA scraper should not be considered reliable merely because it runs. It should be considered reliable when the data it produces continues to satisfy the contract it was built to provide.\nProject\nThe complete project, implementation, deployment configuration, benchmarks, tests, and local setup are available in the\nScraper-CI GitHub repository\n.\n🎬 Watch the demos\n▶\n3-minute Scraper-CI demo on YouTube\n▶\n7-minute technical deep dive on YouTube\n🚀 Live applications\nScraper-CI Control Plane\nFlight Intelligence\n🔧 Backend\nAPI\nHealth Check\nOpenAPI / Swagger\nThe two application links above are the primary entry points for the project. The backend is the shared control-plane API behind the Scraper-CI UI and the downstream consumer. If a Render service is sleeping or temporarily unavailable, a live URL may briefly return an availability error until the service wakes.\nAI / coding-assistant disclosure\nThis project was developed with assistance from\nChatGPT and OpenAI Codex\n. AI assistance was used for coding support, implementation iteration, debugging, and development assistance. The system architecture, source-profiling model, crawl-policy design, acquisition abstraction, reliability model, recovery logic, verification lifecycle, downstream consumer architecture, and infrastructure decisions were designed, reviewed, integrated, and verified as part of the project development process. AI-assisted code was treated as implementation assistance rather than as an authority over the system design.\nFinal takeaway\nScraper-CI started with a simple observation: a scraper can succeed technically while failing silently in the data it produces.\nThe solution wasn't to build another scraper. It was to build the layer that makes acquisition\nobservable, bounded, recoverable, and trustworthy\n.\nThat is the idea behind Scraper-CI.\n#\nweb-scraping\n#\nweb-development\n#\ndata-engineering\n#\nsoftware-development",
  "metadata": {
    "title": "Scraper-CI: Web Data Reliability with Bright Data",
    "description": "How I built Scraper-CI, a web data reliability control plane for profiling, governing, validating, diagnosing, and recovering web acquisition.",
    "keywords": null,
    "author": null,
    "og:title": "Scraper-CI: Web Data Reliability with Bright Data",
    "og:description": "How I built Scraper-CI, a web data reliability control plane for profiling, governing, validating, diagnosing, and recovering web acquisition.",
    "og:url": "https://jaivalsuthar.hashnode.dev/scraper-ci-web-data-reliability-control-plane",
    "og:site_name": "Jaival Suthar | Engineering Portfolio",
    "og:image": "https://cdn.hashnode.com/uploads/og-images/6972fc71207454ddbf7903dd/bf71ae57-ebc7-46c7-83b5-8ff7f0c61a7f.png",
    "og:image:width": "1200",
    "og:image:height": "630",
    "og:type": "article",
    "twitter:card": "summary_large_image",
    "twitter:creator": "https://x.com/Jaival_Suthar",
    "twitter:title": "Scraper-CI: Web Data Reliability with Bright Data",
    "twitter:description": "How I built Scraper-CI, a web data reliability control plane for profiling, governing, validating, diagnosing, and recovering web acquisition.",
    "twitter:image": "https://cdn.hashnode.com/uploads/og-images/6972fc71207454ddbf7903dd/bf71ae57-ebc7-46c7-83b5-8ff7f0c61a7f.png",
    "article:published_time": "2026-08-23T05:40:59.685Z",
    "article:modified_time": "2026-08-23T05:40:59.685Z",
    "article:author": "https://jaivalsuthar.hashnode.dev/@JaivalSuthar"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/4a51670fa0c2/code/body-000.txt",
      "chars": 50,
      "language": "text"
    },
    {
      "file": "attachments/4a51670fa0c2/code/body-001.txt",
      "chars": 19,
      "language": "text"
    },
    {
      "file": "attachments/4a51670fa0c2/code/body-002.txt",
      "chars": 107,
      "language": "text"
    },
    {
      "file": "attachments/4a51670fa0c2/code/body-003.txt",
      "chars": 58,
      "language": "text"
    },
    {
      "file": "attachments/4a51670fa0c2/code/body-004.txt",
      "chars": 154,
      "language": "text"
    },
    {
      "file": "attachments/4a51670fa0c2/code/body-005.txt",
      "chars": 115,
      "language": "text"
    },
    {
      "file": "attachments/4a51670fa0c2/code/body-006.txt",
      "chars": 211,
      "language": "text"
    },
    {
      "file": "attachments/4a51670fa0c2/code/body-007.txt",
      "chars": 139,
      "language": "text"
    },
    {
      "file": "attachments/4a51670fa0c2/code/body-008.txt",
      "chars": 158,
      "language": "text"
    },
    {
      "file": "attachments/4a51670fa0c2/code/body-009.txt",
      "chars": 78,
      "language": "text"
    },
    {
      "file": "attachments/4a51670fa0c2/code/body-010.txt",
      "chars": 249,
      "language": "text"
    },
    {
      "file": "attachments/4a51670fa0c2/code/body-011.txt",
      "chars": 120,
      "language": "text"
    },
    {
      "file": "attachments/4a51670fa0c2/code/body-012.txt",
      "chars": 119,
      "language": "text"
    },
    {
      "file": "attachments/4a51670fa0c2/code/body-013.txt",
      "chars": 115,
      "language": "text"
    },
    {
      "file": "attachments/4a51670fa0c2/code/body-014.txt",
      "chars": 198,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/uploads/covers/6972fc71207454ddbf7903dd/2e1c633c-536b-4e2c-9480-5f61df5213c1.png",
      "bytes": 1602123,
      "reused": true,
      "local_path": "attachments/4a51670fa0c2/images/000.png",
      "src": "https://cdn.hashnode.com/uploads/covers/6972fc71207454ddbf7903dd/2e1c633c-536b-4e2c-9480-5f61df5213c1.png",
      "alt": "Scraper-CI: Building a Web Data Reliability & Crawl-Governance Control Plane With Bright Data",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/uploads/covers/6972fc71207454ddbf7903dd/c6ad15e5-fabd-48b6-8ba2-925e7c858511.png",
      "bytes": 46571,
      "reused": true,
      "local_path": "attachments/4a51670fa0c2/images/001.png",
      "src": "https://cdn.hashnode.com/uploads/covers/6972fc71207454ddbf7903dd/c6ad15e5-fabd-48b6-8ba2-925e7c858511.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/uploads/covers/6972fc71207454ddbf7903dd/8bab2d1e-be68-4ff0-800b-dc900a91183f.png",
      "bytes": 172842,
      "reused": true,
      "local_path": "attachments/4a51670fa0c2/images/002.png",
      "src": "https://cdn.hashnode.com/uploads/covers/6972fc71207454ddbf7903dd/8bab2d1e-be68-4ff0-800b-dc900a91183f.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/uploads/covers/6972fc71207454ddbf7903dd/02fdf390-471e-443a-a50f-85e8d7d2734b.png",
      "bytes": 3678,
      "reused": true,
      "local_path": "attachments/4a51670fa0c2/images/003.png",
      "src": "https://cdn.hashnode.com/uploads/covers/6972fc71207454ddbf7903dd/02fdf390-471e-443a-a50f-85e8d7d2734b.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/uploads/covers/6972fc71207454ddbf7903dd/211dd781-8f59-41ad-9479-d7f0cc8f38bc.png",
      "bytes": 17929,
      "reused": true,
      "local_path": "attachments/4a51670fa0c2/images/004.png",
      "src": "https://cdn.hashnode.com/uploads/covers/6972fc71207454ddbf7903dd/211dd781-8f59-41ad-9479-d7f0cc8f38bc.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/uploads/covers/6972fc71207454ddbf7903dd/52cf6243-73f5-452e-91c2-bdf5fc70efef.png",
      "bytes": 40752,
      "reused": true,
      "local_path": "attachments/4a51670fa0c2/images/005.png",
      "src": "https://cdn.hashnode.com/uploads/covers/6972fc71207454ddbf7903dd/52cf6243-73f5-452e-91c2-bdf5fc70efef.png",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [
    {
      "url": "https://www.youtube.com/watch?v=wOKMk2Btyd8",
      "src": "https://youtu.be/wOKMk2Btyd8",
      "status": "ok",
      "kind": "single_video",
      "local_path": "attachments/4a51670fa0c2/videos/repair-wOKMk2Btyd8.mp4",
      "bytes": 14790084,
      "valid": true,
      "height": 1080,
      "width": 1920,
      "duration": "179.961000",
      "reused": false
    },
    {
      "url": "https://www.youtube.com/watch?v=7tQj2uWBRwM",
      "src": "https://youtu.be/7tQj2uWBRwM",
      "status": "ok",
      "kind": "single_video",
      "local_path": "attachments/4a51670fa0c2/videos/repair-7tQj2uWBRwM.mp4",
      "bytes": 28425901,
      "valid": true,
      "height": 1080,
      "width": 1920,
      "duration": "425.421000",
      "reused": false
    }
  ],
  "links": [
    {
      "href": "https://jaivalsuthar.hashnode.dev/scraper-ci-web-data-reliability-control-plane.md",
      "label": "View as Markdown",
      "type": "a"
    },
    {
      "href": "https://hashnode.com/@JaivalSuthar",
      "label": "J",
      "type": "a"
    },
    {
      "href": "https://x.com/Jaival_Suthar",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://www.linkedin.com/in/jaival-suthar-57195b257/",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://github.com/Jaival-Suthar/sci",
      "label": "https://github.com/Jaival-Suthar/sci",
      "type": "a"
    },
    {
      "href": "https://youtu.be/wOKMk2Btyd8",
      "label": "3-minute Scraper-CI demo on YouTube",
      "type": "a"
    },
    {
      "href": "https://youtu.be/7tQj2uWBRwM",
      "label": "7-minute technical deep dive on YouTube",
      "type": "a"
    },
    {
      "href": "https://scraper-ci-ui.onrender.com/",
      "label": "Scraper-CI Control Plane",
      "type": "a"
    },
    {
      "href": "https://flight-intelligence-rj3l.onrender.com/",
      "label": "Flight Intelligence",
      "type": "a"
    },
    {
      "href": "https://sci-av37.onrender.com/",
      "label": "API",
      "type": "a"
    },
    {
      "href": "https://sci-av37.onrender.com/api/health",
      "label": "Health Check",
      "type": "a"
    },
    {
      "href": "https://sci-av37.onrender.com/docs",
      "label": "OpenAPI / Swagger",
      "type": "a"
    },
    {
      "href": "https://jaivalsuthar.hashnode.dev/tag/web-scraping",
      "label": "# web-scraping",
      "type": "a"
    },
    {
      "href": "https://jaivalsuthar.hashnode.dev/tag/web-development",
      "label": "# web-development",
      "type": "a"
    },
    {
      "href": "https://jaivalsuthar.hashnode.dev/tag/data-engineering",
      "label": "# data-engineering",
      "type": "a"
    },
    {
      "href": "https://jaivalsuthar.hashnode.dev/tag/software-development",
      "label": "# software-development",
      "type": "a"
    }
  ],
  "retry_at": "2026-10-03T08:10:28",
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:51:03Z",
  "source_url": "https://jaivalsuthar.hashnode.dev/scraper-ci-web-data-reliability-control-plane",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 22827,
  "expected_body_images": 6,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/4a51670fa0c2/pages/source.html",
  "local_html": "attachments/4a51670fa0c2/pages/article.html",
  "body_base_path": "attachments/4a51670fa0c2/pages/",
  "body_markdown_path": "attachments/4a51670fa0c2/pages/article.md"
}