Crawl
SpiderPro's core crawling engine efficiently analyzes your website.
Main Features
- Crawl up to 100,000 pages (Pro plan)
- Respect robots.txt optional
- Custom cookies and headers
- Automatic sitemap detection
How to Start a Crawl
- Enter your site's URL in the address bar
- Configure options as needed
- Click "Start Crawl"
Crawl Options
| Option | Description |
|---|---|
| Max pages | Maximum number of pages to crawl |
| JavaScript | Enable JavaScript rendering (Pro) |
| Captures | Save rendered HTML |
| Screenshots | Capture page screenshots (Pro) |
TIP
For JavaScript sites (React, Vue, Angular), enable JavaScript rendering to get complete content.