Crawler

These documents give you an overview of the core Scrapy Cluster Crawler and how to work within both Scrapy and Scrapy Cluster to fit your needs.

Design

Learn about the design considerations for the Scrapy Cluster Crawler

Quick Start

How to use and run the distributed crawlers

Controlling

Learning how to control your Scrapy Cluster will enable you to get the most out of it

Extension

How to use both Scrapy and Scrapy Cluster to enhance your crawling capabilites

Settings

Explains all of the settings used by the Crawler