Managing crawls

Adding knowledge-base sources is covered in Build your knowledge base. This page focuses on the fine-tuning of a crawl: bot-protected sites, interruptions, scope and freshness.

Stealth crawling

If a standard crawl gets a 403 on a page (bot protection like Cloudflare), SemAgent automatically switches to stealth mode: it accesses with a real browser fingerprint (camoufox) through a proxy. This fallback lets you crawl your own protected sites.

Proxy pool

• Stealth requests go through a proxy pool (rotation).
• Proxies are auto-tested hourly; you get an alert if something breaks.
• You can reorder, edit or disable proxies from the panel.

Checkpoint & resume

Long crawls can be interrupted by network/restart. SemAgent keeps a checkpoint: the crawl resumes from the URL it stopped at, no restart. An "orphan watchdog" reaps stuck jobs; there's a timeout ceiling for very long jobs.

Crawl reason-log

For each page, a record of why it was fetched/skipped/errored is kept. "Why wasn't this page indexed?" is always answered here — did a filter not match, did robots block it, was the content empty, did a bot check block it.

Scope & limits

• URL filters (url_include) crawl only the paths you want.
• Page limit caps how many pages a source fetches.
• Fewer pages than expected? The filter may be too narrow or the limit too low — check the reason-log.

Scheduled re-crawl

As your content changes, refresh sources automatically at set intervals; the widget answers with current information.