scrapyd

安装

需求环境

  • >=Python 2.6

  • >=Twisted 8.0

  • >=Scrapy 0.17

通用安装

[root@102 ~]$ pip install scrapyd
Collecting scrapyd
Downloading scrapyd-1.2.0-py2.py3-none-any.whl
Requirement already satisfied: six in /usr/lib/python2.7/site-packages (from scrapyd)
Requirement already satisfied: Twisted>=8.0 in /usr/lib64/python2.7/site-packages (from scrapyd)
Requirement already satisfied: Scrapy>=1.0 in /usr/lib64/python2.7/site-packages (from scrapyd)
Requirement already satisfied: zope.interface>=3.6.0 in /usr/lib64/python2.7/site-packages (from Twisted>=8.0->scrapyd)
Requirement already satisfied: constantly>=15.1 in /usr/lib/python2.7/site-packages (from Twisted>=8.0->scrapyd)
Requirement already satisfied: incremental>=16.10.1 in /usr/lib/python2.7/site-packages (from Twisted>=8.0->scrapyd)
Requirement already satisfied: Automat>=0.3.0 in /usr/lib/python2.7/site-packages (from Twisted>=8.0->scrapyd)
Requirement already satisfied: hyperlink>=17.1.1 in /usr/lib/python2.7/site-packages (from Twisted>=8.0->scrapyd)
Requirement already satisfied: PyDispatcher>=2.0.5 in /usr/lib/python2.7/site-packages (from Scrapy>=1.0->scrapyd)
Requirement already satisfied: lxml in /usr/lib64/python2.7/site-packages (from Scrapy>=1.0->scrapyd)
Requirement already satisfied: service-identity in /usr/lib/python2.7/site-packages (from Scrapy>=1.0->scrapyd)
Requirement already satisfied: w3lib>=1.17.0 in /usr/lib/python2.7/site-packages (from Scrapy>=1.0->scrapyd)
Requirement already satisfied: pyOpenSSL in /usr/lib64/python2.7/site-packages (from Scrapy>=1.0->scrapyd)
Requirement already satisfied: parsel>=1.1 in /usr/lib/python2.7/site-packages (from Scrapy>=1.0->scrapyd)
Requirement already satisfied: cssselect>=0.9 in /usr/lib/python2.7/site-packages (from Scrapy>=1.0->scrapyd)
Requirement already satisfied: queuelib in /usr/lib/python2.7/site-packages (from Scrapy>=1.0->scrapyd)
Requirement already satisfied: setuptools in /usr/lib/python2.7/site-packages (from zope.interface>=3.6.0->Twisted>=8.0->scrapyd)
Requirement already satisfied: attrs in /usr/lib/python2.7/site-packages (from Automat>=0.3.0->Twisted>=8.0->scrapyd)
Requirement already satisfied: pyasn1 in /usr/lib/python2.7/site-packages (from service-identity->Scrapy>=1.0->scrapyd)
Requirement already satisfied: pyasn1-modules in /usr/lib/python2.7/site-packages (from service-identity->Scrapy>=1.0->scrapyd)
Installing collected packages: scrapyd
Successfully installed scrapyd-1.2.0

Note

如果提示python.h文件文件缺失或者gcc命令不存在,请安装开发环境yum groupinstall “development tools”。

开发工程

API接口

配置文件