SemAgentBuild your knowledge base
Your widget answers only from your knowledge base. This guide shows how to build one from scratch: by crawling your website, adding forum and file sources, keeping it fresh and raising quality.
Goal: When the widget answers, it doesn't make things up — it cites the source it found, and says "I don't know" when it can't (strict-KB grounding). A good knowledge base means good answers, so source selection and the quality loop matter.
1. Add your website (crawl)
The fastest path: give your site URL and start a crawl. SemAgent discovers pages two ways:
- Sitemap — if your site has a
sitemap.xml, all listed URLs are fetched (the cleanest method). - BFS link following — with no sitemap, it starts from your URL and expands through internal links breadth-first.
In the knowledge base panel, choose Add source → Website and enter the start URL. To narrow scope, use url_include patterns — e.g. to crawl only the help center:
Start URL: https://example.com/
Include only these paths (url_include):
/help/
/faq/
/blog/Start narrow. Crawl help/FAQ pages first, check answer quality, then widen. Crawling the whole site at once can add noise (menus, footers, campaign pages).
2. Add a forum source (optional)
If you run a community forum, SemAgent auto-detects 5 forum engines: phpBB, inSided, Discourse, XenForo, vBulletin. Add the forum URL as a source; the engine is recognized and threads are parsed.
- Keep the answered threads only (
only_answered) filter on — unresolved/spam threads shouldn't pollute the knowledge base. - Forum content is often the most valuable source: real customer questions with real solutions.
3. Upload files
For content that isn't on the web — product manuals, policy documents, price lists — upload files directly. Supported types: PDF, Word, plain text. Uploaded files are chunked and indexed just like any other source.
4. Start the crawl and monitor it
After adding sources, start the crawl. During and after it you can watch the crawl log — each line shows a page and its reason:
| Status | Meaning |
|---|---|
| Fetched | Page indexed. |
| Skipped | Out of scope (url_include didn't match), duplicate, or empty content. |
| Error | Unreachable (404, timeout, bot block) — the reason is written down. |
"Why wasn't this page indexed?" is always answered in this log. If a long crawl is interrupted, checkpoint/resume lets it continue from where it stopped — no restart from scratch.
5. Bot-protected sites (stealth)
If your own site sits behind a bot protection like Cloudflare, a standard crawl may get a 403. In that case SemAgent switches to stealth crawling: it accesses with a real browser fingerprint (camoufox) through a proxy pool.
Stealth is only for crawling sites you own. Don't use it to bypass someone else's bot protection — that violates terms of service.
6. Keep it fresh (scheduled re-crawl)
As your content changes, the knowledge base goes stale. Use scheduled re-crawl to refresh sources automatically at set intervals (e.g. weekly). That way the widget always gives current price/policy/product info.
7. Raise quality (the loop)
A knowledge base isn't set-and-forget; it's fed. Two tools make the loop easy:
- Answer test pipeline — run sample questions before going live; see which answer came from which source.
- Unanswered-question report — the real customer questions the widget couldn't answer collect in a panel. This list is your knowledge base roadmap: add a source for each gap.
Healthy loop: Publish → read the unanswered-question report → add missing content / crawl a source → test again. Within a few weeks your answer rate climbs noticeably.
8. Organize (KB folders)
Once many sources pile up, group them with folders (e.g. "Products", "Return policy", "FAQ"). Folders are organizational — they don't change search/retrieval behavior, just management.
Next steps
- Tool reference — how
kb_searchand grounding are used on the agent side. - Widget setup & persona (soon) — connecting your knowledge base to the live widget.
- Answer quality & grounding (soon) — the internals of chunking, hybrid search and strict-KB.