Streamline the creation of web scraping spiders with minimal manual coding. Ideal for teams and organizations with recurring data extraction needs.
THIS LIBRARY IS HIGHLY EXPERIMENTAL
Extracting data with LLMs can be expensive. SpiderCreator offers an alternative where LLMs are only used during the spider creation process, and the spiders themselves run on traditional browser automation methods, lowering recurring costs for repeated scraping tasks.
With SpiderCreator, you can automate large parts of the web scraping setup without deep expertise in Playwright or complex DOM manipulation. Ideal for developers and technical teams that need affordable, repeatable spider generation workflows.
SpiderCreator grew out of a long-running interest in AI-assisted web extraction. Its roots go back to 2019, after exploring early web-wrapper ideas such as MohamedHmini/iww, a project focused on AI-based web mining, web-content extraction, and DOM analysis.
The current project applies that line of experimentation to Playwright-based spiders, using LLMs to assist the creation process while keeping the generated extraction logic reviewable and reusable.
Contributions are welcome. Feel free to open issues or submit PRs for bugs, experiments, documentation improvements, and feature requests on GitHub.
Contribute on GitHub