A technical deep dive into source profiling, bounded crawl policies, acquisition routing, data reliability, diagnosis, self-healing, verification, and downstream intelligence.
I write technical articles that analyze real world engineering problems through the lens of production systems.
My work focuses on architectural tradeoffs, UI layer system design, performance optimization, and reliability patterns derived from shipping and maintaining scalable applications.
A scraper returning HTTP 200 is not necessarily healthy. It can return thousands of records, finish without an exception, and still be producing bad data — a website changes its structure, a field disappears, pagination behaves differently, a page starts returning a different template, a required field goes empty. The scraper keeps running the whole time.
From the infrastructure's perspective, everything looks fine:
HTTP 200
↓
Records returned
↓
Job completed
From the application's perspective:
The data is wrong.
That gap is what I wanted to address with Scraper-CI: a web-data reliability and recovery control plane with Bright Data at its center. The goal was not to build another scraper. The goal was to build the layer around the scraper that can answer:
What does this source look like?
What should actually be crawled?
What should not be crawled?
Which acquisition capability makes sense?
Did the extraction still satisfy its contract?
If the data degraded, why?
Can the system recover?
Did the recovery actually fix the problem?
Can downstream applications consume the resulting data without knowing how it was acquired?
The resulting lifecycle is:
PROFILE
↓
POLICY
↓
ROUTE
↓
ACQUIRE
↓
VALIDATE
↓
DIAGNOSE
↓
HEAL
↓
VERIFY
↓
CONSUME
This article explains how I built that system, why I designed the boundaries this way, and why treating "a scraper ran successfully" and "the scraper's data is trustworthy" as two separate questions turned out to be the central design decision. Consider a scraper that historically produces 100 records at 98% required-field completeness — healthy by any reasonable measure. Then the target website changes its layout, and the same scraper still returns 100 records, still returns HTTP 200, but now sits at 42% completeness. Nothing crashed. The record count didn't even move. But the dataset is clearly degraded, and no infrastructure-level signal would have caught it.
🔗 Scraper-CI GitHub Repository: (https://github.com/Jaival-Suthar/sci)
That led to the central design principle behind Scraper-CI:
A scraper should not be considered reliable merely because it runs. It should be considered reliable when the data it produces continues to satisfy the contract it was built to provide.
Scraper-CI therefore treats acquisition success and data correctness as separate concerns:
Acquisition succeeded
≠
Extraction is trustworthy
That distinction drives the rest of the architecture.
The first architectural decision was to not rebuild the acquisition layer. Bright Data already provides sophisticated web-acquisition infrastructure — Scraper Studio handles navigation, pagination, sitemap loading, browser/code execution, and structured extraction, along with its own self-healing capabilities. That's valuable, mature infrastructure, and Scraper-CI is built to sit directly on top of it rather than compete with it. Bright Data isn't a bolted-on dependency here — it's the acquisition core the rest of the system is designed around.
The boundary is deliberate:
Bright Data
= acquisition (the core)
Scraper-CI
= intelligence + control + reliability + recovery
Consumer
= domain-specific interpretation
This separation became the foundation for everything else.
A website isn't simply a graph of URLs and links pointing to more URLs and links — it carries signals that help determine how it should be approached in the first place. Scraper-CI introduced a source profiling layer for exactly this reason. Given a target URL, the profiler inspects signals such as page type, platform, JavaScript dependency, internal and external links, canonical relationships, pagination, sitemap availability, repeated-content patterns, and domain/page relationships.
The profile isn't a report for its own sake — it becomes input to the next question: what should this acquisition be allowed to do?
A scraper can have powerful navigation primitives and still lack an explicit statement of intent. I wanted Scraper-CI to make crawl scope visible and bounded, so the source profile feeds into an inferred crawl policy:
Internal links ✓
Canonical links ✓
Pagination ✓
Sitemap ✓
Unrelated external ✕
The policy also expresses bounded constraints — depth ≤ 2, pages ≤ 50, no external domains, that kind of thing. The important distinction is that the crawler has capabilities, while the policy defines the intended boundary of those capabilities. That's why I think of this as crawl governance rather than just another crawler implementation.
There's an important distinction worth being precise about here. Bright Data already supports sitemap loading in Scraper Studio, so I wouldn't claim Bright Data lacks sitemap support — that would simply be wrong. Bright Data provides the primitive. Scraper-CI adds a different abstraction around it: sitemap discovery and sitemap-derived crawl scope become part of source intelligence and crawl-policy inference.
Bright Data
↓
Sitemap-loading capability
Scraper-CI
↓
Detect sitemap-related source signals
↓
Use them as crawl intelligence
↓
Build bounded crawl intent
↓
Pass that intent into acquisition
The contribution isn't inventing sitemap support — it's the control-plane abstraction around the primitive.
The inferred crawl policy isn't merely stored as metadata — it influences the acquisition request itself.
User extraction intent
+
Source-derived crawl intelligence
↓
Acquisition description
↓
Bright Data collector
The acquisition intent can carry constraints like depth ≤ 2, pages ≤ 50, internal = 1, canonical = 1, pagination = 1, sitemap = 1, external = 0. This creates two layers of protection: first, the acquisition system receives the intended crawl constraints up front (acquisition guidance); second, Scraper-CI independently evaluates the resulting acquisition against the expected state afterward (independent validation). The policy isn't just documentation — it participates in execution and is subsequently checked.
Once the source has been profiled and the crawl policy inferred, the next question is which acquisition capability should handle this source. This became the Capability Router. Instead of assuming every URL gets the same treatment, the system evaluates source signals and produces an acquisition decision.
Source Profile
│
▼
Capability Router
│
├── primary capability
├── fallback capabilities
├── evidence
└── confidence
A routing decision might look conceptually like: source signals → recommended capability → confidence: 78% → acquisition. That confidence score is deliberately not presented as a guarantee — it means the router found stronger evidence for this strategy, not that the scraper is guaranteed to succeed. Reliability is measured from actual extraction results, not from routing confidence.
Scraper-CI uses an abstraction around acquisition capabilities rather than hard-coding the rest of the control plane to one implementation. The current capability model includes pipeline, unlocker, browser, scraper_studio, and search. Some are concrete execution paths, others are extensible capability contracts — that distinction is intentional, since I didn't want the architecture to pretend an executor exists when it doesn't.
Capability contract
↓
Concrete adapter
↓
Acquisition provider
This means additional acquisition mechanisms can be introduced later without rewriting source profiling, crawl policy, reliability, diagnosis, recovery, verification, or downstream consumers. The result: acquisition can evolve without becoming the architecture.
Once Bright Data returns structured data, Scraper-CI asks a different question: is the result still trustworthy? A simplified pipeline runs schema validation, required-field validation, completeness, uniqueness, record-count drift, record comparison, historical drift, and health scoring — landing on HEALTHY, DEGRADED, or INVALID. The system never treats "records exist" as proof of correctness; it evaluates the records themselves against schema validity, required-field completeness, uniqueness, record-count drift, record-level differences, and historical reliability. The goal throughout is detecting silent degradation — the kind of failure that never throws an exception.
A health score alone isn't enough. If the system reports "Health: 54, Status: DEGRADED," the next question is why. Scraper-CI's diagnosis layer classifies problems: missing required fields point to extraction degradation, schema mismatch points to contract violation, blocked acquisition points to an acquisition problem, high JavaScript dependency points to an execution strategy problem. This creates a clean separation — validation answers what is wrong, diagnosis answers why it's wrong, and the recovery planner answers what to try next. Diagnosis is persisted with the run and becomes input to recovery planning.
A common automation pattern is to retry exactly the same thing after a failure, but that's not always useful. If the problem is a missing field, repeating the same acquisition may just reproduce the same missing field. If acquisition is blocked, the system may need a different capability entirely. If the page is JavaScript-heavy, the execution strategy needs to change. Scraper-CI makes recovery diagnosis-driven:
Diagnosis
│
├── Missing field → Self-healing
├── Schema failure → Self-healing
├── Blocked acquisition → Unlocker strategy
├── JS dependency → Browser strategy
└── Unknown → Fallback strategy
The recovery planner is deterministic and independently testable.
When an acquisition finishes in a degraded or invalid state, Scraper-CI can enter an automatic recovery path. The current workflow allows up to two automatic healing attempts per incident:
Run → DEGRADED/INVALID → Healing attempt 1 → Re-run → Verify
→ still degraded? → Healing attempt 2 → Re-run → Verify
The important part is what happens after healing. A successful repair operation doesn't automatically mean the data is correct — Scraper-CI runs the extraction again and independently evaluates the result. The loop is observe → diagnose → heal → re-run → measure.
I didn't want recovery to become "something is broken, fix it." Instead, Scraper-CI generates a diagnosis-derived repair instruction, carrying target information, observed degradation reasons, current field completeness, the extraction contract, preservation rules, and scope constraints. The repair instructions emphasize preserving existing fields and meanings, not expanding crawl scope, not introducing unrelated navigation, and making the smallest extraction change necessary. The goal is to restore the extraction contract with the smallest reasonable change — a repair that fixes one field by dramatically changing what the scraper crawls isn't actually a good repair.
There's an important distinction between failure recovery and intentional contract evolution. Automatic healing applies when something degraded and the system should restore the existing extraction contract. Manual correction applies when the extraction still works but the desired contract has intentionally changed — for example, adding an author field while preserving the existing extraction. That's not a failure; it's an intentional requirement change. Keeping these separate prevents every change in requirements from being treated as scraper failure.
One of the easiest mistakes in a self-healing system is treating the repair API response as proof that the repair worked. I deliberately avoided that — recovery is incomplete until the resulting extraction has been independently evaluated.
Existing state → Repair → New acquisition → Independent validation
→ Before/after comparison → Verification result
Verification evaluates schema pass, completeness pass, health improvement, regression state, record-level differences, recovery delta, and diagnosis state, landing on either REPAIR VERIFIED or REPAIR REJECTED. This distinction is fundamental: a repair action is not the same thing as a repaired dataset.
All of this eventually needed a UI. The Scraper-CI dashboard is an operator control plane over the backend orchestration, with main actions covering Automated, Inspect, Profile, Status, Reliability, Diagnose, Correct data, Verify, and Open URL. These aren't independent frontend implementations — the frontend calls the backend API, which invokes the same domain and orchestration logic used by the CLI. The UI is a control surface, not the system itself.
Inspect answers which acquisition capability to use — it exposes source intelligence, recommended capability, confidence, supporting evidence, and fallback capabilities. This is the routing view. Profile answers what the source looks like and what crawl boundaries can be inferred — it exposes source signals, relationships, sitemap information, pagination, internal/external structure, and the inferred crawl policy. This is the source-intelligence view. Keeping these separate makes the architecture understandable to an operator.
A single run tells you whether something worked today. It doesn't tell you whether the scraper is reliable. Scraper-CI therefore persists historical reliability state — run health, health score, record count, validation state, degradation, recovery attempts, verification state — turning reliability into a historical property rather than a single status field. That lets "did the latest run pass?" and "has this target been stable over time?" be answered separately.
To prove Scraper-CI wasn't just an abstract control plane, I built a concrete downstream application: Flight Intelligence.
Public aviation source → Bright Data → Scraper-CI → Validation
→ Trusted structured data → Flight Intelligence
Flight Intelligence doesn't need to understand the acquisition mechanics — how Bright Data collected the records, which capability was selected, how the collector was created, how crawl policy was inferred, how diagnosis works, or how healing works. It just consumes structured data. That separation was intentional.
Flight Intelligence is domain-specific — it understands aircraft. Scraper-CI doesn't, and shouldn't.
Scraper-CI
│ structured snapshot
▼
Flight Intelligence
├── aircraft counts
├── altitude intelligence
├── speed intelligence
├── aircraft identity
└── historical activity
The consumer derives observations like aircraft count, airborne versus ground activity, average and maximum altitude and speed, altitude bands, high-altitude traffic, new and disappeared aircraft, cumulative distinct aircraft, and historical activity — none of which needs to leak back into Scraper-CI. That's what makes the consumer replaceable.
The larger idea isn't "I built a flight dashboard" — it's that I built a reusable reliability layer that can feed different applications. Today it's Scraper-CI feeding Flight Intelligence. Tomorrow it could just as easily feed monitoring, analytics, research, alerting, RAG, agent workflows, or any other domain-specific application, without rebuilding the acquisition and reliability infrastructure each time.
The deployed system is split into independent services:
The backend owns the Bright Data credential; the frontend never receives the API key. Persistent control-plane state lives in PostgreSQL. Flight Intelligence's production collection is scheduled separately through GitHub Actions, running every two hours (0 */2 * * *) — so the consumer isn't backed by a static demo dataset, but by a live, recurring pipeline: scheduled workflow → scrape-ci run → Bright Data → Scraper-CI → PostgreSQL → Flight Intelligence.
Web acquisition has a hard testing problem: the web itself is nondeterministic. A live website can change between two test runs, making it difficult to tell whether a failure came from the system or from the external source changing underneath it. Scraper-CI includes deterministic tests and benchmark scenarios covering acquisition behavior, API contracts, health scoring, completeness degradation, schema validation, capability routing, crawl policy inference, relationship classification, diagnosis, drift, recovery planning, recovery guards, reliability history, and repair verification — the goal being to keep the intelligence and reliability layers reproducible even when the external web isn't.
1) HTTP success is a weak signal. An HTTP response tells you a request succeeded, not that the extracted dataset is correct — and that single fact changes the architecture considerably.
2) Crawl scope should be explicit. Once a system starts following links, "what does this scraper crawl?" becomes an architectural question. Encoding scope as a policy makes that intent visible, testable, and enforceable.
3) Don't confuse primitives with policy. Bright Data already provides powerful crawling and acquisition primitives. The interesting problem is deciding which primitives to use, how to bound them, what evidence supports the decision, and whether the resulting acquisition satisfied the intended contract.
4) Recovery without verification is dangerous. A healing operation can report success while the resulting data remains wrong. Heal ≠ verify — recovery must end with independent verification.
5) Keep domain intelligence downstream. Aircraft-specific logic doesn't belong in a generic scraping reliability layer, and neither does ecommerce, jobs, news, or finance logic. Scraper-CI produces trusted structured snapshots; consumers interpret them. That boundary is what makes the system reusable.
Worth stating explicitly: Scraper-CI does not claim to have invented web scraping, sitemap parsing, pagination, browser automation, anti-bot infrastructure, proxy infrastructure, scraper self-healing, or structured extraction. Bright Data already provides significant, mature infrastructure in all of these areas — it's the acquisition core this entire system is built to sit on top of. The architectural contribution is the layer around it: source profiling, crawl policy, acquisition routing, reliability evaluation, diagnosis, recovery planning, and independent verification. The project is stronger for being honest about that boundary.
This is the system I ended up building.
The web is not a static API. Selectors change, schemas change, page relationships change, navigation changes, content changes. And the most dangerous failures are often not crashes — they're successful executions that produce increasingly incorrect data. That's why web acquisition needs a control plane that treats reliability as a lifecycle rather than a boolean: understand → bound → route → acquire → validate → diagnose → recover → verify → consume.
Bright Data is the acquisition core at the heart of this system. Scraper-CI provides the intelligence, control, reliability, and recovery around it. Flight Intelligence demonstrates what can be built on top. That separation is ultimately the point.
A scraper should not be considered reliable merely because it runs. It should be considered reliable when the data it produces continues to satisfy the contract it was built to provide.
The complete project, implementation, deployment configuration, benchmarks, tests, and local setup are available in the Scraper-CI GitHub repository.
▶ 3-minute Scraper-CI demo on YouTube
▶ 7-minute technical deep dive on YouTube
The two application links above are the primary entry points for the project. The backend is the shared control-plane API behind the Scraper-CI UI and the downstream consumer. If a Render service is sleeping or temporarily unavailable, a live URL may briefly return an availability error until the service wakes.
This project was developed with assistance from ChatGPT and OpenAI Codex. AI assistance was used for coding support, implementation iteration, debugging, and development assistance. The system architecture, source-profiling model, crawl-policy design, acquisition abstraction, reliability model, recovery logic, verification lifecycle, downstream consumer architecture, and infrastructure decisions were designed, reviewed, integrated, and verified as part of the project development process. AI-assisted code was treated as implementation assistance rather than as an authority over the system design.
Scraper-CI started with a simple observation: a scraper can succeed technically while failing silently in the data it produces.
The solution wasn't to build another scraper. It was to build the layer that makes acquisition observable, bounded, recoverable, and trustworthy.
That is the idea behind Scraper-CI.