【阿旭机器学习实战】【27】贝叶斯模型：新闻分类实战----CounterVecorizer与TfidVectorizer构建特征向量对比

【阿旭机器学习实战】系列文章主要介绍机器学习的各种算法模型及其实战案例，欢迎点赞，关注共同学习交流。

本文介绍了新闻分类实战案例，并通过两种方法CounterVecorizer与TfidVectorizer构建特征向量，然后分别建模对比。

1. 导入数据并查看信息

from sklearn.datasets import fetch_20newsgroups
from sklearn.model_selection import train_test_split
1
2

# 加载新闻数据
news = fetch_20newsgroups(subset='all')
1
2

# data为一个列表，长度18846，每一个元素为一个新闻内容的字符串
print(len(news.data))
1
2

18846
1

news.data[0]
1

"From: Mamatha Devineni Ratnam \nSubject: Pens fans reactions\nOrganization: Post Office, Carnegie Mellon, Pittsburgh, PA\nLines: 12\nNNTP-Posting-Host: po4.andrew.cmu.edu\n\n\n\nI am sure some bashers of Pens fans are pretty confused about the lack\nof any kind of posts about the recent Pens massacre of the Devils. Actually,\nI am  bit puzzled too and a bit relieved. However, I am going to put an end\nto non-PIttsburghers' relief with a bit of praise for the Pens. Man, they\nare killing those Devils worse than I thought. Jagr just showed you why\nhe is much better than his regular season stats. He is also a lot\nfo fun to watch in the playoffs. Bowman should let JAgr have a lot of\nfun in the next couple of games since the Pens are going to beat the pulp out of Jersey anyway. I was very disappointed not to see the Islanders lose the final\nregular season game.          PENS RULE!!!\n\n"
1

# news.target为目标分类对应的编号
news.target
1
2

array([10,  3, 17, ...,  3,  1,  7])
1

# 目标标签名称有20个，因此一共分20类新闻
len(news.target_names)
1
2

20
1

# 查看第一篇新闻属于什么类别
print(news.target[0])
print(news.target_names[news.target[0]])
1
2
3

10
rec.sport.hockey
1
2

2. 使用CountVectorizer构建单词字典并建模预测

CountVectorizer方法构建单词的字典，每个单词实例被转换为特征向量的一个数值特征，每个元素是特定单词在文本中出现的次数

2.1 CountVectorizer用法示例

from sklearn.feature_extraction.text import CountVectorizer

texts=["pig bird cat","dog dog cat cat","bird fish bird", 'pig bird']
cv = CountVectorizer()
# 将文本向量化
cv_fit=cv.fit_transform(texts)

# 查看转换后的向量，会统计单词个数，并写在指定索引位置
print(cv.get_feature_names())   # 获取单词序列
print(cv_fit.toarray())         # 将文本变为向量
1
2
3
4
5
6
7
8
9
10

['bird', 'cat', 'dog', 'fish', 'pig']
[[1 1 0 0 1]
 [0 2 2 0 0]
 [2 0 0 1 0]
 [1 0 0 0 1]]
1
2
3
4
5

2.2 使用CountVectorizer进行特征向量转换

cv = CountVectorizer()
cv_data = cv.fit_transform(news.data)
1
2

2.3 使用贝叶斯模型进行建模预测

from sklearn.model_selection import cross_val_score 
from sklearn.naive_bayes import MultinomialNB


x_train,x_test,y_train,y_test = train_test_split(cv_data, news.target)

mul_nb = MultinomialNB()

train_scores = cross_val_score(mul_nb, x_train, y_train, cv=3, scoring='accuracy')  
test_scores = cross_val_score(mul_nb, x_test, y_test, cv=3, scoring='accuracy')  
print("train scores:", train_scores)
print("test scores:", test_scores)
1
2
3
4
5
6
7
8
9
10
11
12

train scores: [0.81457936 0.81260611 0.82925792]
test scores: [0.64258555 0.56687898 0.61700767]
1
2

3. 使用TfidfVectorizer进行特征向量转换并建模预测

TfidfVectorizer使用了一个高级的计算方法，称为Term Frequency Inverse Document Frequency (TF-IDF)。IDF是逆文本频率指数(Inverse Document Frequency)。

TFIDF的主要思想是：如果某个词或短语在一篇文章中出现的频率TF高，并且在其他文章中很少出现，则认为此词或者短语具有很好的类别区分能力，适合用来分类。

它一个衡量一个词在文本或语料中重要性的统计方法。直觉上讲，该方法通过比较在整个语料库的词的频率，寻求在当前文档中频率较高的词。这是一种将结果进行标准化的方法，可以避免因为有些词出现太过频繁而对一个实例的特征化作用不大的情况(我猜测比如a和and在英语中出现的频率比较高，但是它们对于表征一个文本的作用没有什么作用)。

3.1 TfidfVectorizer使用示例

from sklearn.feature_extraction.text import TfidfVectorizer
# 文本文档列表
text = ["The quick brown fox jumped over the lazy dog.",
"The lazy dog.",
"The brown fox"]
# 创建变换函数
vectorizer = TfidfVectorizer()
# 词条化以及创建词汇表
vectorizer.fit(text)
# 总结
print(vectorizer.vocabulary_)
print(vectorizer.idf_)
# 编码文档
vector = vectorizer.transform([text[0]])
# 总结编码文档
print(vector.shape)
print(vector.toarray())
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

{'the': 7, 'quick': 6, 'brown': 0, 'fox': 2, 'jumped': 3, 'over': 5, 'lazy': 4, 'dog': 1}
[1.28768207 1.28768207 1.28768207 1.69314718 1.28768207 1.69314718
 1.69314718 1.        ]
(1, 8)
[[0.29362163 0.29362163 0.29362163 0.38607715 0.29362163 0.38607715
  0.38607715 0.45604677]]
1
2
3
4
5
6

3.2 对新闻数据进行TfidfVectorizer变换

# 创建变换函数
vectorizer = TfidfVectorizer()
# 词条化以及创建词汇表
tfidf_data = vectorizer.fit_transform(news.data)
1
2
3
4

3.3 进行建模与预测

x_train,x_test,y_train,y_test = train_test_split(tfidf_data, news.target)

mul_nb = MultinomialNB()
train_scores = cross_val_score(mul_nb, x_train, y_train, cv=3, scoring='accuracy')  
test_scores = cross_val_score(mul_nb, x_test, y_test, cv=3, scoring='accuracy')  
print("train scores:", train_scores)
print("test scores:", test_scores)
1
2
3
4
5
6
7

train scores: [0.8238287  0.83379325 0.81937952]
test scores: [0.68103995 0.68809675 0.68030691]
1
2

3.4 去除停用词并进行建模与预测

def get_stop_words():
    result = set()
    for line in open('stopwords_en.txt', 'r').readlines():
        result.add(line.strip())
    return result

# 加载停用词
stop_words = get_stop_words()
# 创建变换函数
vectorizer = TfidfVectorizer(stop_words=stop_words)

# 词条化以及创建词汇表
tfidf_data = vectorizer.fit_transform(news.data)

x_train,x_test,y_train,y_test = train_test_split(tfidf_data,news.target)

mul_nb = MultinomialNB(alpha=0.01)

train_scores = cross_val_score(mul_nb, x_train, y_train, cv=3, scoring='accuracy')  
test_scores = cross_val_score(mul_nb, x_test, y_test, cv=3, scoring='accuracy')  
print("train scores:", train_scores)
print("test scores:", test_scores)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

train scores: [0.90419669 0.89577584 0.90095643]
test scores: [0.85107731 0.8433121  0.84526854]
1
2

通过对比发现使用 TfidVectorizer构建特征向量的建模效果要好于CounterVecorizer。同时去除停用词之后，模型准确率也会有较大的提升。

如果内容对你有帮助，感谢点赞+关注哦！

欢迎关注我的公众号:阿旭算法与机器学习，共同学习交流。
更多干货内容持续更新中…