python中Request Payload参数使用（持续更新）

在post请求中的参数中，Request Payload会经常碰到，但很多同学可能并未真正如何去使用，碰到复杂点的Request Payload往往一头雾水，下面就我日常所遇到的一些案例做个简单的说明，希望对大家有帮助。

关于Form Data与Request Payload的区别，可以参考文章：https://juejin.cn/post/6844904149809627149
https://leejason.blog.csdn.net/article/details/88720102

1 正常的json参数（非Request Payload）

这种是最普通常见的POST请求类型，请求头中Content-Type值通常为：application/x-www-form-urlencoded，也就是所谓的Form Data方式。
在scrapy的用法如下：

param = {"playerid": "25635555"}
return scrapy.FormRequest(url=self.url_post,
                          formdata=param,
                          meta={
                                'gameUserName': seed_keyword, #自定义参数
                                'SpiderRunTime': SpiderRunTime,#自定义参数
                                  },
                          dont_filter=True,
                          callback=self.parse)
1
2
3
4
5
6
7
8
9

在post中用法
requests.post(self.url_post, data=param )

2 简单Request Payload

如果发送失败的话，记得添加method=“POST”

param = {"playerid": "25635555"}
return scrapy.FormRequest(url=self.url_post,
                          body=json.dumps(param),#字典转换为json
                          meta={
                              'gameUserName': seed_keyword, #自定义参数
                              'SpiderRunTime': SpiderRunTime,#自定义参数
                          },
                          dont_filter=True,
                          callback=self.parse)
1
2
3
4
5
6
7
8
9

3 复杂Request Payload

如果是复杂Request Payload，比如Request Payload是列表或或有子字典等，简单用json.dumps(param)转换后请求会经常报错。且使用Scrapy框架时利用自带的FormReuqest 是无法完成要求的。一般用python requests，直接将内容转换为字符串提交。如下：

处理这个参数一般有两种方式：
A：字典转字符

headers = {
    "Accept": "application/json, text/plain, */*",
    "Content-Type": "application/json;charset=UTF-8;",
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36",
}
data = {
    "data": {
        "mainSearchPcConditionForm": {
            "city": "410",
            "dq": "410",
            "pubTime": "",
            "currentPage": 0,
            "pageSize": 40,
            "key": "",
            "suggestTag": "",
            "workYearCode": "",
            "compId": "",
            "compName": "",
            "compTag": "",
            "industry": "",
            "salary": "",
            "jobKind": "",
            "compScale": "",
            "compKind": "",
            "compStage": "",
            "eduLevel": ""
        },
        "passThroughForm": {
            "scene": "init",
            "ckId": "rdfpkd81uti5ryk4h8paa16kszjlk0kb",
            "suggest": None # 这里需要注意的是，在浏览器中我们看到这字段的值是null，由于python语法没有null类型，只能None。有些人喜欢用第三方工具从浏览器搬迁，比如用curl2py生成的结果，是‘’‘"suggest": null‘’‘,需要手动修改为"suggest": None,如果修改成"suggest": "null"，转换后发送请求会报错的。
        }
    }
}

data = json.dumps(data)
res = requests.post(
    "https://apic.liepin.com/api/com.liepin.searchfront4c.pc-search-job",
    headers=headers,
    data=data
)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

B：直接构造字符串
如下：

data = '''
    {"data":{"mainSearchPcConditionForm":{"city":"410","dq":"410","pubTime":"","currentPage":0,"pageSize":40,"key":"","suggestTag":"","workYearCode":"0","compId":"","compName":"","compTag":"","industry":"","salary":"","jobKind":"","compScale":"","compKind":"","compStage":"","eduLevel":""},"passThroughForm":{"scene":"init","ckId":"hnk5a1i55f1uelq0oqzjjxkp0edju1eb","suggest":null}}}
'''
res = requests.post(
    "https://apic.liepin.com/api/com.liepin.searchfront4c.pc-search-job",
    headers=headers,
    data=data
)
print(res.text)
print(res.status_code)
1
2
3
4
5
6
7
8
9
10