scrapy snippet
2024-10-21 11:48:55
1. spider文件
from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
from scrapy.selector import HtmlXPathSelector item = DomzItem()
image_urls = hxs.select('//img/@src').extract()
item['image_urls'] = ["http:" + x for x in image_urls]
return item
from scrapy.selector import HtmlXPathSelector
hxs = HtmlXPathSelector(response)
class MySpider(CrawlSpider): #控制下载速度
name = 'myspider'
download_delay = 2
$ scrapy crawl somespider -s JOBDIR=crawls/somespider-1
#这样开始下载之后可以Ctrl + C停止,恢复下载还是同样的命令
$ scrapy crawl somespider -s JOBDIR=crawls/somespider-1
name = "wikipedia"
allowed_domains = ["wikipedia.org"]
start_urls = [
"http://en.wikipedia.org/wiki/Pune"
]
2. setting文件
ITEM_PIPELINES = ['scrapy.contrib.pipeline.images.ImagesPipeline']
IMAGES_STORE= '...'
3. item 文件
image_urls = Field()
images = Field()
最新文章
- 动态给textView加图片
- 门面模式的典型应用 Socket 和 Http(post,get)、TCP/IP 协议的关系总结
- JAVA - 优雅的记录日志(log4j实战篇)
- centos Linux 统计某个文件夹占用空间大小
- Cheatsheet: 2014 05.01 ~ 05.31
- Redis 2:简单使用
- HTTP协议学习笔记-1
- linux route命令学习
- Java Web中资源的访问路径
- 普通table表格样式及代码大全(全)
- Android异步请求
- How and when: ridge regression with glmnet
- Project入门学习
- Redis做分布式锁
- 记一次用express手写博客
- ElasticSearch 评分排序
- Webpack代理proxy配置,解决本地跨域调试问题,同时允许绑定host域名调试
- IO中File类基本使用
- JDK 之 NIO 2 WatchService、WatchKey(监控文件变化)
- vscode 学习笔记 —— 调试 (以 node 为例)