cli-toolstools
crawler
Web crawling and scraping reference — robots.txt protocol, Scrapy framework, anti-bot detection, headless browsers, and legal considerations
maintainer
openclaw
Updated 3/24/2026
Stars
4001
Forks
1095
quick start
Installation and usage
Web crawling and scraping reference — robots.txt protocol, Scrapy framework, anti-bot detection, headless browsers, and legal considerations
Installation
$ install --globalskills.sh
Usage
Once installed, you can use this skill by running the following command in your terminal:
skills use crawler