How FDEs Build Reliable Web Agents

Your Web Agent Needs a Trust Layer, Not a Bigger Model

Part 2 of 2 in the web data agent series. Part 1 built an agent that could pull useful data from the web. Part 2 is about the harder problem: deciding what the agent should believe before anyone acts on it.

In Part 1, we built a sourcing agent. It reads the web, finds companies that look like design-partner candidates, and returns a list with names, contacts, emails, and supporting links. On a demo run, it works. The rows look clean. The formatting is good. The agent produces something that looks like a useful spreadsheet.

Then you run it on the real web for a week.

That is when the field reality shows up. Some companies are no longer active. Some contacts left months ago. Some emails bounce. Some pages were stale. Some fields came from the wrong part of the page. One “company name” is actually a navigation label the scraper grabbed from the header.

The agent does not flag any of this. It hands you the bad rows with the same confidence as the good rows.

That is the real failure. Not that the agent made mistakes. Every system that reads the open web will make mistakes. The failure is that the agent had no way to know which rows were safe to act on.

A clean-looking spreadsheet is not a reliable system. It is just a liability with nice formatting.

For a Forward Deployed Engineer, this is where the real work starts. The job is not to make the demo prettier. The job is to make the system safe enough to use in the field. That means turning a fuzzy customer need into an operating bar.

For this build, the bar is simple:

Produce a weekly list of qualified design-partner candidates where at least 95 percent of accepted rows are real, current, and supported by evidence, under $50 per week, with no manual cleanup before outreach.

That bar changes the architecture. A web search API is not enough. A scraper is not enough. A model that writes clean rows is not enough. You need a trust layer.

The trust layer sits between “the agent found something” and “a person or system acts on it.” It answers one question: should we believe this record enough to use it?

The real question is what happens after the agent acts

Most agent demos stop at output. An FDE cannot stop there.

In the field, the important question is not whether the agent produced a list. The important question is what action someone will take because of that list.

For a sourcing agent, the action is outreach. That means a bad row has real consequences. A wrong email can bounce. A stale contact wastes time. A bad-fit company makes the founder look careless. A made-up field can turn into an embarrassing message. If this runs every week, those errors compound.

So the agent should not simply return fifty companies. It should return a decision report: which companies are safe to act on, which were rejected, which need review, and why.

That is the difference between an agent that generates output and a system that supports a business workflow.

Separate truth from fit

The first design decision is to separate truth from fit.

Truth asks whether a claim is real, current, and supported by evidence. Fit asks whether that true record matches the customer profile.

“Is this company still active?” is a truth question. “Is this company a good design partner for our product?” is a fit question. “Does this person work there?” is a truth question. “Is this the right buyer persona?” is a fit question.

Do not mix these layers. If you mix them, you build a one-off lead-generation tool that may work for one customer and one workflow. If you keep them separate, the trust layer becomes reusable. You can use the same layer for a research agent, a price-monitoring agent, a market-map agent, a competitive-intelligence agent, or any web data agent that turns messy pages into structured claims.

This is one of the core FDE lessons in the build: build the reusable reliability layer first, then put the customer-specific logic on top.

Every field needs evidence

A value by itself cannot be trusted.

This is not enough:

{
  "company": "Acme AI",
  "contact": "Jane Smith",
  "email": "jane@acme.ai"
}

That record looks useful, but it is not auditable. Where did the company name come from? Where did the email come from? When was the page fetched? Was the page saved? Did the extractor pull the value from the right section? Was this value found directly, inferred, guessed, or generated?

Without that information, the system cannot defend the row.

A better record carries evidence with every field:

class FieldEvidence(BaseModel):
    value: str | int | None
    source_url: str
    fetched_at: datetime
    snapshot_id: str
    extractor: str
    confidence: float

class Candidate(BaseModel):
    company: FieldEvidence
    contact: FieldEvidence
    email: FieldEvidence
    size: FieldEvidence

Now each field carries a trail: the value, the source URL, the time it was fetched, the saved page snapshot, the extractor that produced it, and a confidence score.

The snapshot_id matters. Save the raw page once, key it by a hash of the page content, and store that hash. This lets you re-check any claim later against the exact page the agent saw. It also lets you cache verification results. If the same page appears again and the page hash has not changed, you do not need to pay to verify it again.

The extractor field is useful for operations. When a selector breaks, or an LLM extractor starts pulling the wrong value, failures often cluster around one extractor. That gives you a debugging handle.

Without provenance, the output is just a list. With provenance, the output becomes auditable.

The FDE standard is simple: if you cannot trace a field back to evidence, you cannot defend it in front of a customer.

The four checks

The trust layer runs four checks: provenance and shape, support, recency, and consistency. Run them in that order because the cheapest checks should run first. Do not spend tokens on problems that code can catch.

Each check produces a field-level verdict. Field verdicts roll up into a record-level verdict. The record gets one of three decisions: accept, reject, or review.

That third option is critical. Real field systems are full of uncertainty. If the only options are accept and reject, the agent will guess. Guessing is how bad rows get shipped. Review is not a failure. Review is how the system stays honest.

Check 1: provenance and shape

The first check is plain code. Before asking a model to reason, ask basic questions. Did the fetch succeed? Was the final URL expected? Was the page large enough to be real content? Did we get a captcha, login wall, block page, or empty render? Does the value have the right shape?

This catches obvious failures. “Home” is not a company name. “Login” is not a contact. “Menu” is probably not a company. A page title saying “Are you human?” should not be treated as evidence. A 200 response with almost no content may be a block page. A headcount of 80 million is probably not a startup.

This check is fast, deterministic, and cheap. Run it on every field.

def check_shape(field_name: str, value: str) -> Verdict:
    if field_name == "email" and not looks_like_email(value):
        return reject("email shape failed")

    if field_name == "company" and value.lower() in ["home", "login", "menu"]:
        return reject("company looks like navigation text")

    return accept("shape passed")

This work is not glamorous, but it saves the rest of the system. The FDE rule is: spend code before tokens.

Check 2: support

Support is the most important check in the trust layer. It asks whether the source page actually supports the claim.

Not whether the page contains similar words. Actually supports it.

Suppose the claim is: “Acme AI has 180 employees.” A weak check searches the page for “180.” That is not enough. The page might say 180 reviews, 180 customers, founded in 2018, or “we help companies get a 180-degree customer view.” The number appearing on the page does not mean the page supports the claim.

The better method is entailment. Take the page text and the claim, then ask a verifier whether the page entails the claim, contradicts the claim, or says nothing useful.

Contradiction is a hard reject. Neutral is also unsafe. If the page does not support the claim, the agent should not act as if it does.

def supports(claim: str, page_text: str) -> SupportResult:
    """
    Returns:
    - entail
    - contradict
    - neutral

    Also returns the exact span of page text used as evidence.
    """
    ...

The span matters. If the verifier says the page supports the claim, it should return the exact text it relied on. That span is what the human reviewer sees. It is also how you debug the verifier when it gets something wrong.

This is where the trust layer earns its keep. A sourcing agent without support checks is formatting guesses. A sourcing agent with support checks can say: this contact came from this source, fetched on this date, and this exact span supports the claim.

That is a different product.

The FDE rule is: do not verify vibes. Verify claims against saved evidence.

Check 3: recency

Some fields rot quickly. Others barely rot at all.

An email can go stale in months. A title can go stale in months. A company size may be good for a longer period. A founding year usually does not go stale.

So do not use one freshness rule for the whole record. Set freshness budgets per field.

FRESHNESS_DAYS = {
    "email": 90,
    "contact": 90,
    "title": 90,
    "size": 365,
    "founded": None
}

If email evidence is older than 90 days, refresh it or send it to review. If founding-year evidence is old, that may be fine.

This is a simple check, but it changes system behavior. The agent no longer treats old evidence and fresh evidence as equal.

The FDE rule is: freshness is field-specific, not record-specific.

Check 4: consistency

When two sources disagree, do not silently pick one.

If one source says the company has 180 employees and another says 1,200, that disagreement is a signal. The trust layer should mark the field for review.

def check_consistency(values: list[FieldEvidence]) -> Verdict:
    if values_disagree_strongly(values):
        return review("sources disagree")
    return accept("sources consistent")

Consistency checks are only possible when you have more than one source. If you only have one source, record that too. A single-sourced field may still be acceptable, but the reviewer should know it is single-sourced.

The FDE rule is: disagreement is data. Do not hide it.

The verdict is the product

After the checks run, the system rolls field verdicts into a record verdict.

If a required field is contradicted, reject the record. If a required field is unsupported, reject or review it. If required evidence is stale, refresh or review it. If sources strongly disagree, review it. If all required fields pass, accept it.

For outreach, the required fields are usually the company, the contact, the email, evidence that the person is relevant, and evidence that the company fits the target profile. Do not over-verify fields that do not affect the action. If the action is sending an email, the contact and email deserve stronger checks than the founding year.

The final output should not be “here are 50 companies.” It should be:

50 companies reviewed. 32 verified. 18 dropped or held for review: 6 stale, 5 poor fit, 4 unsupported contacts, 3 scraping errors. Safe to act on: 32.

That is field-ready. A founder can understand it. A sales team can use it. A customer can trust it.

Human review is part of the architecture

The trust layer should not pretend every case can be automated. Some records will be uncertain. That is normal.

The key is to make review fast. For each review item, show the field value, source URL, saved page snapshot, exact supporting span, failed checks, reason for review, and one-click accept, reject, or edit.

A good review queue takes minutes. A bad review queue takes hours and gets bypassed.

This is where FDE judgment matters. “Human in the loop” is not a checkbox. It is a workflow design problem. If review is too slow, people will ignore it. If people ignore it, the trust layer stops mattering.

How you know it works

You cannot evaluate this system by looking at one good run. You need a fixed test set.

Take 30 to 50 records. Save the raw pages. Label the correct verdicts. Include clean examples and broken examples. Break some on purpose: stale contact, fake email, blocked page, wrong company name, unsupported headcount, contact pulled from the wrong page section, or two sources with conflicting size numbers.

This is your golden set.

Every time you change the trust layer, run the golden set again. Track two numbers: bad-record recall and accepted-record precision. Bad-record recall asks: of the truly bad records, how many did the system catch? Accepted-record precision asks: of the records the system accepted, how many were actually good?

You need both. A system that rejects everything catches every bad record, but it is useless. A system that accepts everything keeps volume high, but it lets bad data through.

For this use case, the operating floor might be: catch at least 85 percent of known bad records, keep accepted-row precision above 95 percent, and never accept a required claim that failed support.

Then wire the golden set into CI. If a change fixes one example but breaks five others, you want to know before it ships.

The golden set should grow over time. Every real miss becomes a test case. Every customer complaint becomes a regression test. Every weird page the web throws at you becomes part of the system memory.

Over time, the eval set becomes one of the most valuable assets in the build.

The FDE rule is: the eval set is not a side artifact. It is the operating memory of the field system.

How to keep it affordable

The expensive part is not fetching pages. The expensive part is verification.

A support check can require a model call per claim. If you verify every field on every record with a large model, the bill grows fast. Four moves keep cost under control.

First, verify only action-critical fields. For outreach, verify the contact and email carefully. Verify company fit enough to avoid obvious misses. Do not spend the same verification budget on fields that do not change the action.

Second, cache by snapshot hash. If the page content has not changed, the support verdict may not need to be recomputed. Store verification results by snapshot hash, claim, model version, and trust-layer version. When the same page and claim appear again, reuse the verdict.

Third, use a cheap verifier before a larger judge. Most claims are easy. Use a small, narrow verifier first. Escalate only uncertain cases to a larger model.

Fourth, batch where possible. If your provider allows it, verify multiple claims for one record in a single call. Avoid one call per tiny field when one structured call can do the job.

The FDE rule is: reliability that cannot fit the customer’s cost budget is not production-ready.

The FDE case file

The customer does not need “more leads.” The customer needs a weekly list of design-partner candidates that is safe enough to use for outreach without manual cleanup. The measurable bar is 95 percent accepted-row precision, evidence on every accepted row, under $50 per week, review only for uncertain cases, and no unsupported required claims accepted.

The key architecture decision is to buy the data plane and build the trust layer. Search and scraping are commodities. The customer-specific value is deciding which rows are safe to act on. The system separates truth from fit, stores evidence per field, checks support using entailment, uses three verdicts, runs cheap checks first, and measures itself with a golden set.

The production failure modes are predictable. Companies shut down. Contacts leave. Emails disappear. Selectors start pulling navigation text. Captchas get parsed as content. Login walls look like pages. Empty JavaScript renders slip through. Sources disagree. Caches go stale. Prompts change. Verifiers become too permissive.

The operating dashboard should track fetch success rate, blocked-page rate, extraction failure rate, schema-validation failure rate, support-check pass rate, review rate, accepted-row precision, bad-record recall, cost per verified row, latency per record, cache hit rate, top rejection reasons, and failures by extractor.

Watch the trends. A rising review rate means the agent is becoming less certain. A sudden cache hit-rate drop may mean pages changed or a model version changed. A cluster of failures from one extractor means a source layout probably moved. A cost spike means verification volume is escaping the budget.

The FDE rule is: if you cannot see the system drift, you cannot operate it.

What you keep

This started as a web sourcing agent. What you keep is bigger than that.

You now have a reusable trust layer for web data agents. Records carry evidence. Fields are checked before action. Support is verified against saved pages. Stale evidence is treated differently by field. Source disagreement triggers review. Uncertain cases escalate instead of being guessed. The golden set catches regressions. Cost is controlled through gating and caching.

The scraper may change. The customer profile may change. The model may change. The trust layer is the part that travels.

That is the real lesson.

A good field system does not just produce output. It earns the right to act.

For web agents, that right comes from one thing:

A trust layer that decides what to believe before the agent does anything with it.