SemAgentBuild your knowledge base
Your widget answers only from your knowledge base. This guide shows how to build one from scratch: by crawling your website, adding forum and file sources, keeping it fresh and raising quality.
Goal: When the widget answers, it doesn't make things up — it cites the source it found, and says "I don't know" when it can't (strict-KB grounding). A good knowledge base means good answers, so source selection and the quality loop matter.
1. Add your website (crawl)
The fastest path: give your site URL and start a crawl. SemAgent discovers pages two ways:
- Sitemap — if your site has a
sitemap.xml, all listed URLs are fetched (the cleanest method). - BFS link following — with no sitemap, it starts from your URL and expands through internal links breadth-first.
In the knowledge base panel, choose Add source → Website and enter the start URL. To narrow scope, use url_include patterns — e.g. to crawl only the help center:
Start URL: https://example.com/
Include only these paths (url_include):
/help/
/faq/
/blog/Start narrow. Crawl help/FAQ pages first, check answer quality, then widen. Crawling the whole site at once can add noise (menus, footers, campaign pages).
2. Add a forum source (optional)
If you run a community forum, SemAgent auto-detects 5 forum engines: phpBB, inSided, Discourse, XenForo, vBulletin. Add the forum URL as a source; the engine is recognized and threads are parsed.
- Keep the answered threads only (
only_answered) filter on — unresolved/spam threads shouldn't pollute the knowledge base. - Forum content is often the most valuable source: real customer questions with real solutions.
3. Upload files
For content that isn't on the web — product manuals, policy documents, price lists — upload files directly. Supported types: PDF, Word (.docx), Excel (.xlsx), CSV, JSON, Markdown, plain text and images (PNG, JPG, WebP; the text in them is read). Up to 10 MB per file. Uploaded files are chunked and indexed just like any other source. Every file passes a virus scan after upload; a harmful file is removed and written to the Audit Log. Once its text has been extracted the file itself isn't kept; its text stays in your knowledge base.
4. Start the crawl and monitor it
After adding sources, start the crawl. During and after it you can watch the crawl log — each line shows a page and its reason:
| Status | Meaning |
|---|---|
| Fetched | Page indexed. |
| Skipped | Out of scope (url_include didn't match), duplicate, or empty content. |
| Error | Unreachable (404, timeout, bot block) — the reason is written down. |
"Why wasn't this page indexed?" is always answered in this log. If a long crawl is interrupted, checkpoint/resume lets it continue from where it stopped — no restart from scratch.
5. Bot-protected sites (stealth)
If your own site sits behind a bot protection like Cloudflare, a standard crawl may get a 403. In that case SemAgent switches to stealth crawling: it accesses with a real browser fingerprint (camoufox) through a proxy pool.
Stealth is only for crawling sites you own. Don't use it to bypass someone else's bot protection — that violates terms of service.
6. Keep it fresh (scheduled re-crawl)
As your content changes, the knowledge base goes stale. Use scheduled re-crawl to refresh sources automatically at set intervals (e.g. weekly). That way the widget always gives current price/policy/product info.
7. Raise quality (the loop)
A knowledge base isn't set-and-forget; it's fed. Two tools make the loop easy:
- Answer test pipeline — run sample questions before going live; see which answer came from which source.
- Unanswered-question report — the real customer questions the widget couldn't answer collect in a panel. This list is your knowledge base roadmap: add a source for each gap.
Healthy loop: Publish → read the unanswered-question report → add missing content / crawl a source → test again. Within a few weeks your answer rate climbs noticeably.
8. Organize (KB folders)
Once many sources pile up, group them with folders (e.g. "Products", "Return policy", "FAQ"). Folders are organizational — they don't change search/retrieval behavior, just management.
Next steps
- Widget setup & persona — connecting your knowledge base to the live widget.
- Answer quality & grounding — the internals of chunking, hybrid search and strict-KB.