爬虫2：html页面+beautifulsoap模块+post方式+demo

　　爬取html页面，有时需要设置参数post方式请求，生成json，保存文件中。

1）引入模块

import requests

from bs4 import BeautifulSoup

url_ = 'http://www.c.....................'

2）设置参数

 datas = {

        'yyyy':'',

        'mm':'-12-31',

        'cwzb':"incomestatements",

        'button2':"%CC%E1%BD%BB",

    }

3）post请求

r = requests.post(url,data = datas)

4）设置编码

r.encoding = r.apparent_encoding

5）BeautifulSoup解析request请求

soup = BeautifulSoup(r.text)

6）find_all筛选

soup.find_all('strong',text=re.compile(u"股票代码"))[].parent.contents[]

7）css选择select

soup.select("option[selected]")[].contents[]

beautifulsoap的API请查看　　https://www.crummy.com/software/BeautifulSoup/bs4/doc/index.zh.html#beautifulsoup

Demo：文件读url，设置参数，post方式，beautifulsoap解析，生成json，保存文件中

import requests

from bs4 import BeautifulSoup

import re

import json

import time

fd = open(r"E:\aa.txt","r")

mylist = []

for line in fd:

    mylist.append(line)

url_pre = 'http://www.c.....................'

code = open(r"E:\a---.txt", "a")

for index in xrange(,len(mylist)):

    print index

    url_id = mylist[index].split('?')[-]

    url_id = url_id[-:-]

    datas = {

        'yyyy':'',

        'mm':'-12-31',

        'cwzb':"incomestatements",'button2':"%CC%E1%BD%BB",

    }

    url = url_pre + str(url_id)

    print url

    print datas

    r = requests.post(url,data = datas)

    r.encoding = r.apparent_encoding

    print r

    soup = BeautifulSoup(r.text)

    r.encoding = r.apparent_encoding

    soup = BeautifulSoup(r.text)

    if len(soup.find_all("td",text=re.compile(u"营业收入"))) == :

        continue

    jsonMap = {}

    jsonMap[u'股票代码'] = soup.find_all('strong',text=re.compile(u"股票代码"))[].parent.contents[]

    jsonMap[u'股票简称'] = soup.find_all('strong',text=re.compile(u"股票代码"))[].parent.contents[]

    jsonMap[u'年度'] = soup.select("option[selected]")[].contents[]

    jsonMap[u'报告期'] = soup.select("option[selected]")[].contents[]

    yysr = soup.find_all("td",text=re.compile(u"营业收入"))[].parent

    yysrsoup = BeautifulSoup(str(yysr))

    jsonMap[u'营业收入'] = yysrsoup.find_all('td')[].contents[]

    yylr = soup.find_all("td",text=re.compile(u"营业利润"))[].parent

    yylrsoup = BeautifulSoup(str(yylr))

    jsonMap[u'营业利润'] = yylrsoup.find_all('td')[].contents[]

    strJson = json.dumps(jsonMap, ensure_ascii=False)

    print strJson

    #code.write(strJson)

    code.write(strJson.encode('utf-8') + '\n')

    time.sleep(0.1)

    code.flush()

巴特西

爬虫2：html页面+beautifulsoap模块+post方式+demo

最新文章

热门文章