Pagesift
Self-hosted web scraper, crawler and document parser for AI agents: URLs, PDFs with OCR and Office files to clean Markdown/JSON. Firecrawl- and Jina-compatible…
It is self-hosted and runs as a single Rust binary offering a CLI, Firecrawl and Jina Reader APIs, an MCP server, and SDKs for multiple languages. Its DataGraph feature tracks citations, provenance, version lineage, field-level evidence, and cross-source conflicts. PageSift supports crawling entire sites via sitemaps, trap handling, robots.txt, and per-host limits, outputting WARC files.
Summary drafted from the project's own website. Every sentence is backed by text on that page and was reviewed before publishing.