What's New
Scrapion is a Python web scraping automation library with intelligent fallback mechanisms and automatic browser setup.
Features
- Dual Input Modes: Accept direct URLs or search queries via DuckDuckGo
- Smart URL Management: Automatically splits search results into main (1-5) and backup (6-10) lists
- Intelligent Fallback: Retries with backup URLs if primary URLs fail
- Auto Browser Setup: Firefox browser automatically installs on first use (via Playwright)
- Content Extraction: Robust web content retrieval and conversion to markdown
- Structured Reports: JSON-formatted reports with success/failure tracking
- Flexible API: Use as Python library or CLI tool