【ML特征工程】第 4 章 ：特征缩放的影响：从词袋到 Tf-Idf

示例 4-2。创建平衡分类数据集

# Create a class-balanced subsample to play with
>>> nightlife = \
...   twobiz_reviews[twobiz_reviews.apply(lambda x: 'Nightlife' in x['categories'],
...                                         axis=1)]
>>> restaurants = \
...   twobiz_reviews[twobiz_reviews.apply(lambda x: 'Restaurants' in x['categories'],
...                                         axis=1)]
>>> nightlife_subset = nightlife.sample(frac=0.1, random_state=123)
>>> restaurant_subset = restaurants.sample(frac=0.021, random_state=123)
>>> combined = pd.concat([nightlife_subset, restaurant_subset])
# Split into training and test datasets
>>> training_data, test_data = modsel.train_test_split(combined, 
...                                                    train_size=0.7, 
...                                                    random_state=123)
>>> training_data.shape
(29264, 5)
>>> test_data.shape
(12542, 5)

示例 4-3。变换特征

# Represent the review text as a bag-of-words 
>>> bow_transform = text.CountVectorizer()
>>> X_tr_bow = bow_transform.fit_transform(training_data['text'])
>>> X_te_bow = bow_transform.transform(test_data['text'])
>>> len(bow_transform.vocabulary_)
46924
 
>>> y_tr = training_data['target']
>>> y_te = test_data['target']
 
# Create the tf-idf representation using the bag-of-words matrix
>>> tfidf_trfm = text.TfidfTransformer(norm=None)
>>> X_tr_tfidf = tfidf_trfm.fit_transform(X_tr_bow)
>>> X_te_tfidf = tfidf_trfm.transform(X_te_bow)
 
# Just for kicks, l2-normalize the bag-of-words representation
>>> X_tr_l2 = preproc.normalize(X_tr_bow, axis=0)
>>> X_te_l2 = preproc.normalize(X_te_bow, axis=0)

测试集上的特征缩放

关于特征缩放的一个微妙点是它需要知道特征统计那我们最在实践中可能不知道，例如均值、方差、文档频率，ℓ 2范数等。为了计算 tf-idf 表示，我们必须根据训练数据计算逆文档频率，并使用这些统计数据来缩放训练数据和测试数据。在 scikit-learn 中，将特征转换器拟合到训练数据上相当于收集相关统计数据。然后可以将安装的变压器应用于测试数据。

图 4-3。sigmoid 函数的图示

示例 4-4。使用默认参数训练逻辑回归分类器

>>> def simple_logistic_classify(X_tr, y_tr, X_test, y_test, description):
...     ### Helper function to train a logistic classifier and score on test data
...     m = LogisticRegression().fit(X_tr, y_tr)
...     s = m.score(X_test, y_test)
...     print ('Test score with', description, 'features:', s)
...     return m
 
>>> m1 = simple_logistic_classify(X_tr_bow, y_tr, X_te_bow, y_te, 'bow')
>>> m2 = simple_logistic_classify(X_tr_l2, y_tr, X_te_l2, y_te, 'l2-normalized')
>>> m3 = simple_logistic_classify(X_tr_tfidf, y_tr, X_te_tfidf, y_te, 'tf-idf')
Test score with bow features: 0.775873066497
Test score with l2-normalized features: 0.763514590974
Test score with tf-idf features: 0.743182905438

 矛盾的是，结果表明最准确的分类器是使用 BoW 特征的分类器。这是出乎意料的。事实证明，原因是分类器没有很好地“调整”，这是比较分类器时常见的错误。

重要：比较模型时调整超参数

在比较模型或特征时调整超参数至关重要。软件包的默认设置总是会返回一个模型。但除非软件在后台执行自动调整，否则它很可能会返回基于次优超参数设置的次优模型。分类器性能对超参数设置的敏感性取决于模型和训练数据的分布。逻辑回归对超参数设置相对稳健（或不敏感）。即便如此，有必要找到并使用正确范围的超参数。否则，一种模型相对于另一种模型的优势可能完全取决于调整参数，而不会反映模型或特征的实际行为。

即使是最好的自动调谐包仍然需要指定搜索的上限和下限，而找到这些限制可能需要手动尝试几次。

通过重采样估计方差

现代统计方法假设基础数据来自随机分布。源自数据的模型的性能测量也受到随机噪声的影响。在这种情况下，基于可比较统计数据的数据集，不仅进行一次测量，而且进行多次测量始终是一个好主意。这为我们提供了测量的置信区间。

k折交叉验证就是这样一种策略。重采样是另一种从同一基础数据集生成多个小样本的技术。有关重采样的更多详细信息，请参见 Zheng (2015)。

示例 4-5。使用网格搜索调整逻辑回归超参数

>>> import sklearn.model_selection as modsel
# Specify a search grid, then do a 5-fold grid search for each of the feature sets
>>> param_grid_ = {'C': [1e-5, 1e-3, 1e-1, 1e0, 1e1, 1e2]}
# Tune classifier for bag-of-words representation
>>> bow_search = modsel.GridSearchCV(LogisticRegression(), cv=5, 
...                                  param_grid=param_grid_)
>>> bow_search.fit(X_tr_bow, y_tr)
# Tune classifier for L2-normalized word vector
>>> l2_search = modsel.GridSearchCV(LogisticRegression(), cv=5,
...                                 param_grid=param_grid_)
>>> l2_search.fit(X_tr_l2, y_tr)
# Tune classifier for tf-idf
>>> tfidf_search = modsel.GridSearchCV(LogisticRegression(), cv=5,
...                                    param_grid=param_grid_)
>>> tfidf_search.fit(X_tr_tfidf, y_tr)
# Let's check out one of the grid search outputs to see how it went
>>> bow_search.cv_results_
{'mean_fit_time': array([  0.43648252,   0.94630651,  
          5.64090128,  15.31248307,  31.47010217,  42.44257565]),
'mean_score_time': array([ 0.00080056,  0.00392466,  0.00864897,  0 .00784755,  
         0.01192751,  0.0072515 ]),
'mean_test_score': array([ 0.57897075,  0.7518111 ,  0.78283898,  0.77381766,  
         0.75515992,  0.73937261]),
'mean_train_score': array([ 0.5792185 ,  0.76731652,  0.87697341,  0.94629064,  
         0.98357195,  0.99441294]),
'param_C': masked_array(data = [1e-05 0.001 0.1 1.0 10.0 100.0],
              mask = [False False False False False False],
        fill_value = ?),
'params': ({'C': 1e-05},
  {'C': 0.001},
  {'C': 0.1},
  {'C': 1.0},
  {'C': 10.0},
  {'C': 100.0}),
'rank_test_score': array([6, 4, 1, 2, 3, 5]),
'split0_test_score': array([ 0.58028698,  0.75025624,  0.7799795 ,  0.7726341 ,  
         0.75247694,  0.74086095]),
'split0_train_score': array([ 0.57923964,  0.76860316,  0.87560871,  0.94434003,  
         0.9819308 ,  0.99470312]),
'split1_test_score': array([ 0.5786776 ,  0.74628396,  0.77669571,  0.76627371,  
         0 .74867589,  0.73176149]),
'split1_train_score': array([ 0.57917218,  0.7684849 ,  0.87945837,  0.94822946,  
         0.98504976,  0.99538678]),
'split2_test_score': array([ 0.57816504,  0.75533914,  0.78472578,  0.76832394,  
         0.74799248,  0.7356911 ]),
'split2_train_score': array([ 0.57977019,  0.76613558,  0.87689548,  0.94566657,  
         0.98368288,  0.99397719]),
'split3_test_score': array([ 0.57894737,  0.75051265,  0.78332194,  0.77682843,  
         0.75768968,  0.73855092]),
'split3_train_score': array([ 0.57914745,  0.76678626,  0.87634546,  0.94558346,  
         0.98385443,  0.99474628]),
'split4_test_score': array([ 0.57877649,  0.75666439,  0.78947368,  0.78503076,  
         0.76896787,  0.75      ]),
'split4_train_score': array([ 0.57876303,  0.7665727 ,  0.87655903,  0.94763369,  
         0.98334188,  0.99325132]),
'std_fit_time': array([ 0.03874582,  0.02297261,  1.18862097,  1.83901079,  
         4.21516797,  2.93444269]),
'std_score_time': array([ 0.00160112,  0.00605009,  0.00623053,  0.00698687,  
         0.00713112,  0.00570195]),
'std_test_score': array([ 0.00070799,  0.00375907,  0.00432957,  0.00668246,  
         0.00612049]),
'std_train_score': array([ 0.00032232,  0.00102466,  0.00131222,  0.00143229,  
         0.00100223,  0.00073252])}
# Plot the cross validation results in a box-and-whiskers plot to 
# visualize and compare classifier performance
>>> search_results = pd.DataFrame.from_dict({
...                               'bow': bow_search.cv_results_['mean_test_score'],
...                               'tfidf': tfidf_search.cv_results_['mean_test_score'],
...                               'l2': l2_search.cv_results_['mean_test_score']
...                      })
# Our usual matplotlib incantations. Seaborn is used here to make
# the plot pretty.
>>> import matplotlib.pyplot as plt
>>> import seaborn as sns
>>> sns.set_style("whitegrid")
>>> ax = sns.boxplot(data=search_results, width=0.4)
>>> ax.set_ylabel('Accuracy', size=14)
>>> ax.tick_params(labelsize=14)

图 4-4。每个特征集和正则化设置下分类器准确度的分布——准确度是通过 5 折交叉验证的平均准确度来衡量的

示例 4-6。比较不同特征集的最后训练和测试步骤

# Train a final model on the entire training set, using the best hyperparameter 
# settings found previously. Measure accuracy on the test set.
>>> m1 = simple_logistic_classify(X_tr_bow, y_tr, X_te_bow, y_te, 'bow', 
...                               _C=bow_search.best_params_['C'])
>>> m2 = simple_logistic_classify(X_tr_l2, y_tr, X_te_l2, y_te, 'l2-normalized', 
...                               _C=l2_search.best_params_['C'])
>>> m3 = simple_logistic_classify(X_tr_tfidf, y_tr, X_te_tfidf, y_te, 'tf-idf', 
...                               _C=tfidf_search.best_params_['C'])
Test score with bow features: 0.78360708021
Test score with l2-normalized features: 0.780178599904
Test score with tf-idf features: 0.788470738319

适当的调整提高了所有特征集的准确性，现在这三个特征集在正则化逻辑回归下都产生了相似的分类准确性。tf-idf 模型的准确度分数略高，但差异可能不具有统计显着性。这些结果完全令人费解。如果特征缩放并不比普通的词袋更好，那为什么还要这样做呢？如果 tf-idf 什么都不做，为什么要大声疾呼？我们将在下一节探讨这些问题的答案。