一个简单的异步爬虫

好久没写过爬虫了，今天看到有群友在问就看了下他的代码，大概是一个异步的爬取影视作品链接的爬虫，看到有别的大佬给他解决了问题后他发了张存入mongodb的截图，于是我也正好学习也复习下asyncio。

motor

Motor 提供了一个基于协程的 API，用于从Tornado或asyncio对 MongoDB 的非阻塞访问。Motor 在连接到 MongoDB 或执行 I/O 时从不阻塞事件循环。Motor 封装了几乎所有 PyMongo 的 API 并使其成为非阻塞的。

简单的介绍

代码

import re

import asyncio
import aiohttp
from lxml import etree
import motor.motor_asyncio


client = motor.motor_asyncio.AsyncIOMotorClient()
collection = client.film.kkxz_test


async def index_page(url):
    page_url_list = []
    async with aiohttp.ClientSession() as session:
        async with session.get(url) as req:
            res = await req.json()
            for page_url in res['data']:
                page_url_list.append('https://kkxz.vip/d/' + page_url['attributes']['slug'])

    await save_data(page_url_list)


async def save_data(urls):
    for url in urls:
        async with aiohttp.ClientSession() as session:
            async with session.get(url) as req:
                text = await req.text()
                movie_link = re.findall('href="(https://pan.quark.cn.*?)"', text)
                movie_title = re.findall('
(.*?)
', text)
                print(movie_title[0], movie_link)
                await collection.insert_one({'title': movie_title[0],'link': movie_link})


async def main():
    url = 'https://kkxz.vip/api/discussions?include=user%2ClastPostedUser%2CfirstPost%2Ctags%2Ctags.parent&filter%5Btag%5D=tv&sort&page%5Boffset%5D={id}'
    tasks = [asyncio.create_task(index_page(url.format(id = i))) for i in range(0, 20, 30)]
    await asyncio.wait(tasks)


if __name__ == '__main__':
    asyncio.run(main())

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43