Your First Crawl
Let's run your first SEO crawl with SpiderPro.
Step 1: Enter the URL
In the URL field at the top of the application, enter your website's URL:
https://example.comTIP
Enter the main domain URL. SpiderPro will automatically discover all pages from there.
Step 2: Configure Basic Settings
For your first crawl, the default settings should work fine. Key settings include:
| Setting | Default | Description |
|---|---|---|
| Max Pages | 2000 | Maximum pages to crawl |
| JavaScript | Off | Enable for SPA/React sites |
| Respect robots.txt | On | Follow robots.txt rules |
Step 3: Start the Crawl
Click the Start Crawl button (green play icon).
You'll see:
- Real-time page counter
- Live results appearing in the table
- Progress percentage
Step 4: Analyze Results
Once complete, explore the different views:
Table View
- Filter by status code (200, 301, 404...)
- Sort by any column
- Export to CSV/Excel
Issues View
- Critical errors (red)
- Warnings (yellow)
- Recommendations (blue)
Graph View
- 3D visualization of your site structure
- Identify orphan pages
- See link flow
Common First Crawl Issues
Crawl is too slow
- Reduce concurrent requests in settings
- Check your internet connection
Pages are missing
- Enable JavaScript rendering (Pro)
- Check robots.txt exclusions
Too many 404s
- Normal for broken links
- Review and fix important ones
Next Steps
- Crawl Settings - Fine-tune your crawl
- JavaScript Rendering - For React/Vue sites
- Exports - Save and share results