• Kaggle 专利匹配比赛赛后总结

    比赛简介

    在专利匹配数据集中，选手需要判断两个短语的相似度，一个是anchor ，一个是target
    ，然后输出两者在不同语义(context)的相似度，范围是0-1，我们队伍id为xlyhq，a榜rank 13，b榜ran12，非常感谢@heng zheng、@pythonlan,@leolu1998,@syzong四位队友的努力和付出，最后比较幸运的狗到金牌。

    和其他前排核心思路差不多，我们在这里主要分享下我们的比赛历程以及相关实验的具体结果，以及有意思的尝试

    文本处理

    数据集主要有anchor、target和context字段，另外有额外的文本拼接信息，在比赛过程中我们主要是尝试了以下拼接的尝试：

    • v1:test['anchor'] + '[SEP]' + test['target'] + '[SEP]' + test['context_text']
    • v2:test['anchor'] + '[SEP]' + test['target'] + '[SEP]' +test['context']+ '[SEP]' + test['context_text'],相当于直接把A47类似编码拼接上去
    • v3:test['text'] = test['anchor'] + '[SEP]' + test['target'] + '[SEP]' + test['context'] + '[SEP]' + test['context_text'] 获取更多的文本进行拼接，相当于把A47下面的子类别拼接上去，比如A47B，A47C
    1. context_mapping = {
    2.     "A": "Human Necessities",
    3.     "B": "Operations and Transport",
    4.     "C": "Chemistry and Metallurgy",
    5.     "D": "Textiles",
    6.     "E": "Fixed Constructions",
    7.     "F": "Mechanical Engineering",
    8.     "G": "Physics",
    9.     "H": "Electricity",
    10.     "Y": "Emerging Cross-Sectional Technologies",
    11. }
    12.  
    13. titles = pd.read_csv('./input/cpc-codes/titles.csv')
    14.  
    15.  
    16. def process(text):
    17.     return re.sub(u"\\(.*?\\)|\\{.*?}|\\[.*?]", "", text)
    18.  
    19.  
    20. def get_context(cpc_code):
    21.     cpc_data = titles[(titles['code'].map(len) <= 4) & (titles['code'].str.contains(cpc_code))]
    22.     texts = cpc_data['title'].values.tolist()
    23.     texts = [process(text) for text in texts]
    24.     return ";".join([context_mapping[cpc_code[0]]] + texts)
    25.  
    26.  
    27. def get_cpc_texts():
    28.     cpc_texts = dict()
    29.     for code in tqdm(train['context'].unique()):
    30.         cpc_texts[code] = get_context(code)
    31.     return cpc_texts
    32.  
    33.  
    34. cpc_texts = get_cpc_texts()

    这个拼接方式可以得到不小的提升，但是文本长度变得更长，最大长度设置为300，导致训练更慢

    • v4:核心的拼接方式：test['text'] = test['text'] + '[SEP]' + test['target_info']
    1. # 拼接target info
    2. test['text'] = test['anchor'] + '[SEP]' + test['target'] + '[SEP]' + test['context_text']
    3. target_info = test.groupby(['anchor', 'context'])['target'].agg(list).reset_index()
    4. target_info['target'] = target_info['target'].apply(lambda x: list(set(x)))
    5. target_info['target_info'] = target_info['target'].apply(lambda x: ', '.join(x))
    6. target_info['target_info'].apply(lambda x: len(x.split(', '))).describe()
    7.  
    8. del target_info['target']
    9. test=test.merge(target_info,on=['anchor','context'],how='left')
    10. test['text'] = test['text'] + '[SEP]' + test['target_info'] 
    11. test.head()

    这种拼接方式可以让模型cv和lb分数得到较大提升，通过v3和v4两种不同拼接方式的对比，我们可以发现选取质量更高的文本进行拼接对模型更有提升作用，v3方式中有很多冗余信息，而v4方式中有很多实体级别的关键信息。

    数据划分

    在比赛过程中，我们尝试了不同的数据划分方式，其中包括：

    • StratifiedGroupKFold，这种拼接方式cv与lb线差比较小，分数稍微好一点
    • StratifiedKFold：线下cv比较高
    • 其他Kfold和GrouFold效果不好

    损失函数

    主要可以参考的损失函数有：

    • BCE： nn.BCEWithLogitsLoss(reduction="mean")
    • MSE：nn.MSELoss()
    • Mixture Loss：MseCorrloss 
    1. class CorrLoss(nn.Module):
    2.     """
    3.     use 1 - correlational coefficience between the output of the network and the target as the loss
    4.     input (o, t):
    5.         o: Variable of size (batch_size, 1) output of the network
    6.         t: Variable of size (batch_size, 1) target value
    7.     output (corr):
    8.         corr: Variable of size (1)
    9.     """
    10.     def __init__(self):
    11.         super(CorrLoss, self).__init__()
    12.  
    13.     def forward(self, o, t):
    14.         assert(o.size() == t.size())
    15.         # calcu z-score for o and t
    16.         o_m = o.mean(dim = 0)
    17.         o_s = o.std(dim = 0)
    18.         o_z = (o - o_m)/o_s
    19.  
    20.         t_m = t.mean(dim =0)
    21.         t_s = t.std(dim = 0)
    22.         t_z = (t - t_m)/t_s
    23.  
    24.         # calcu corr between o and t
    25.         tmp = o_z * t_z
    26.         corr = tmp.mean(dim = 0)
    27.         return  1 - corr
    28.     
    29. class MSECorrLoss(nn.Module):
    30.     def __init__(self, p = 1.5):
    31.         super(MSECorrLoss, self).__init__()
    32.         self.p = p
    33.         self.mseLoss = nn.MSELoss()
    34.         self.corrLoss = CorrLoss()
    35.  
    36.     def forward(self, o, t):
    37.         mse = self.mseLoss(o, t)
    38.         corr = self.corrLoss(o, t)
    39.         loss = mse + self.p * corr
    40.         return loss

    我们实验采用的这个损失函数，效果稍微比BCE好一点

    模型设计

    为了提高模型的差异度，我们主要选择了不同模型的变体，其中包括以下五个模型：

    • Deberta-v3-large
    • Bert-for-patents
    • Roberta-large
    • Ernie-en-2.0-Large
    • Electra-large-discriminator

    具体cv分数如下：

    1. deberta-v3-large：[0.8494,0.8455,0.8523,0.8458,0.8658] cv 0.85176
    2. bertforpatents [0.8393, 0.8403, 0.8457, 0.8402, 0.8564] cv 0.8444
    3. roberta-large [0.8183,0.8172,0.8203,0.8193,0.8398] cv 0.8233
    4. ernie-large [0.8276,0.8277,0.8251,0.8296,0.8466] cv 0.8310
    5. electra-large [0.8429,0.8309,0.8259,0.8416,0.846] cv 0.8376

    训练优化

    根据以往比赛经验，我们主要采用了以下模型训练优化方式：

    • 对抗训练：尝试了FGM 对模型训练有提升效果
    1. class FGM():
    2.     def __init__(self, model):
    3.         self.model = model
    4.         self.backup = {}
    5.     def attack(self, epsilon=1., emb_name='word_embeddings'):
    6.         # emb_name这个参数要换成你模型中embedding的参数名
    7.         for name, param in self.model.named_parameters():
    8.             if param.requires_grad and emb_name in name:
    9.                 self.backup[name] = param.data.clone()
    10.                 norm = torch.norm(param.grad)
    11.                 if norm != 0 and not torch.isnan(norm):
    12.                     r_at = epsilon * param.grad / norm
    13.                     param.data.add_(r_at)
    14.     def restore(self, emb_name='emb.'):
    15.         # emb_name这个参数要换成你模型中embedding的参数名
    16.         for name, param in self.model.named_parameters():
    17.             if param.requires_grad and emb_name in name: 
    18.                 assert name in self.backup
    19.                 param.data = self.backup[name]
    20.         self.backup = {}
    • 模型泛化：加入了multidroout
    • ema对模型训练有提升效果
    1. class EMA():
    2.     def __init__(self, model, decay):
    3.         self.model = model
    4.         self.decay = decay
    5.         self.shadow = {}
    6.         self.backup = {}
    7.  
    8.     def register(self):
    9.         for name, param in self.model.named_parameters():
    10.             if param.requires_grad:
    11.                 self.shadow[name] = param.data.clone()
    12.  
    13.     def update(self):
    14.         for name, param in self.model.named_parameters():
    15.             if param.requires_grad:
    16.                 assert name in self.shadow
    17.                 new_average = (1.0 - self.decay) * param.data + self.decay * self.shadow[name]
    18.                 self.shadow[name] = new_average.clone()
    19.  
    20.     def apply_shadow(self):
    21.         for name, param in self.model.named_parameters():
    22.             if param.requires_grad:
    23.                 assert name in self.shadow
    24.                 self.backup[name] = param.data
    25.                 param.data = self.shadow[name]
    26.  
    27.     def restore(self):
    28.         for name, param in self.model.named_parameters():
    29.             if param.requires_grad:
    30.                 assert name in self.backup
    31.                 param.data = self.backup[name]
    32.         self.backup = {}
    33.  
    34. # 初始化
    35. ema = EMA(model, 0.999)
    36. ema.register()
    37.  
    38. # 训练过程中，更新完参数后，同步update shadow weights
    39. def train():
    40.     optimizer.step()
    41.     ema.update()
    42.  
    43. # eval前，apply shadow weights；eval之后，恢复原来模型的参数
    44. def evaluate():
    45.     ema.apply_shadow()
    46.     # evaluate
    47.     ema.restore()

    没有用的尝试：

    • AWP
    • PGD

    模型融合

    根据线下交叉验证分数以及线上分数反馈，我们通过加权融合的方式进行平均融合

    1. from sklearn.preprocessing import MinMaxScaler
    2. MMscaler = MinMaxScaler()
    3. predictions1 = MMscaler.fit_transform(submission['predictions1'].values.reshape(-1,1)).reshape(-1)
    4. predictions2 = MMscaler.fit_transform(submission['predictions2'].values.reshape(-1,1)).reshape(-1)
    5. predictions3 = MMscaler.fit_transform(submission['predictions3'].values.reshape(-1,1)).reshape(-1)
    6. predictions4 = MMscaler.fit_transform(submission['predictions4'].values.reshape(-1,1)).reshape(-1)
    7. predictions5 = MMscaler.fit_transform(submission['predictions5'].values.reshape(-1,1)).reshape(-1)
    8.  
    9.  
    10. # final_predictions=(predictions1+predictions2)/2
    11. # final_predictions=(predictions1+predictions2+predictions3+predictions4+predictions5)/5
    12. # 5：2：1：1：1
    13. final_predictions=0.5*predictions1+0.2*predictions2+0.1*predictions3+0.1*predictions4+0.1*predictions5

    其他尝试

    • two stage
      前期我们做了不同预训练模型的微调，所以特征数量相对较多，我们尝试基于树模型的对文本统计特征以及模型预测做stacking尝试,当时模型是有比较不错的融合效果，下面含有部分代码
    1. # ====================================================
    2. # predictions1
    3. # ====================================================
    4.  
    5. def get_fold_pred(CFG, path, model):
    6.     CFG.path = path
    7.     CFG.model = model
    8.     CFG.config_path = CFG.path + "config.pth"
    9.     CFG.tokenizer = AutoTokenizer.from_pretrained(CFG.path)
    10.     test_dataset = TestDataset(CFG, test)
    11.  
    12.     test_loader = DataLoader(test_dataset,
    13.                              batch_size=CFG.batch_size,
    14.                              shuffle=False,
    15.                              num_workers=CFG.num_workers, pin_memory=True, drop_last=False)
    16.     predictions = []
    17.     for fold in CFG.trn_fold:
    18.         model = CustomModel(CFG, config_path=CFG.config_path, pretrained=False)
    19.         state = torch.load(CFG.path + f"{CFG.model.split('/')[-1]}_fold{fold}_best.pth",
    20.                            map_location=torch.device('cpu'))
    21.         model.load_state_dict(state['model'])
    22.         prediction = inference_fn(test_loader, model, device)
    23.         predictions.append(prediction.flatten())
    24.         del model, state, prediction
    25.         gc.collect()
    26.         torch.cuda.empty_cache()
    27.     # predictions1 = np.mean(predictions, axis=0)
    28.  
    29.     # fea_df = pd.DataFrame(predictions).T
    30.     # fea_df.columns = [f"{CFG.model.split('/')[-1]}_fold{fold}" for fold in CFG.trn_fold]
    31.     # del test_dataset, test_loader
    32.  
    33.     return predictions
    34.  
    35.  
    36. model_paths = [
    37.     "../input/albert-xxlarge-v2/albert-xxlarge-v2/",
    38.     "../input/bert-large-cased-cv5/bert-large-cased/",
    39.     "../input/deberta-base-cv5/deberta-base/",
    40.     "../input/deberta-v3-base-cv5/deberta-v3-base/",
    41.     "../input/deberta-v3-small/deberta-v3-small/",
    42.     "../input/distilroberta-base/distilroberta-base/",
    43.     "../input/roberta-large/roberta-large/",
    44.     "../input/xlm-roberta-base/xlm-roberta-base/",
    45.     "../input/xlmrobertalarge-cv5/xlm-roberta-large/",
    46. ]
    47.  
    48. print("train.shape, test.shape", train.shape, test.shape)
    49. print("titles.shape", titles.shape)
    50.  
    51.  
    52. # for model_path in model_paths:
    53. #     with open(f'{model_path}/oof_df.pkl', "rb") as fh:
    54. #         oof = pickle.load(fh)[['id', 'fold', 'pred']]
    55. # #     oof = pd.read_pickle(f'{model_path}/oof_df.pkl')[['id', 'fold', 'pred']]
    56. #     oof[f"{model_path.split('/')[1]}"] = oof['pred']
    57. #     train = train.merge(oof[['id', f"{model_path.split('/')[1]}"]], how='left', on='id')
    58.     
    59. oof_res=pd.read_csv('../input/train-res/train_oof.csv')
    60.  
    61. train = train.merge(oof_res, how='left', on='id')
    62.  
    63. model_infos = {
    64.     'albert-xxlarge-v2': ['../input/albert-xxlarge-v2/albert-xxlarge-v2/', "albert-xxlarge-v2"],
    65.     'bert-large-cased': ['../input/bert-large-cased-cv5/bert-large-cased/', "bert-large-cased"],
    66.     'deberta-base': ['../input/deberta-base-cv5/deberta-base/', "deberta-base"],
    67.     'deberta-v3-base': ['../input/deberta-v3-base-cv5/deberta-v3-base/', "deberta-v3-base"],
    68.     'deberta-v3-small': ['../input/deberta-v3-small/deberta-v3-small/', "deberta-v3-small"],
    69.     'distilroberta-base': ['../input/distilroberta-base/distilroberta-base/', "distilroberta-base"],
    70.     'roberta-large': ['../input/roberta-large/roberta-large/', "roberta-large"],
    71.     'xlm-roberta-base': ['../input/xlm-roberta-base/xlm-roberta-base/', "xlm-roberta-base"],
    72.     'xlm-roberta-large': ['../input/xlmrobertalarge-cv5/xlm-roberta-large/', "xlm-roberta-large"],
    73. }
    74.  
    75. for model, path_info in model_infos.items():
    76.     print(model)
    77.     model_path, model_name = path_info[0], path_info[1]
    78.     fea_df = get_fold_pred(CFG, model_path, model_name)
    79.     model_infos[model].append(fea_df)
    80.     del model_path, model_name
    81.  
    82. del oof_res

    训练代码：

    1. for fold_ in range(5):
    2.     print("Fold:", fold_)
    3.  
    4.     trn_ = train[train['fold'] != fold_].index
    5.     val_ = train[train['fold'] == fold_].index
    6. #     print(train.iloc[val_].sort_values('id'))
    7.     trn_x, trn_y = train[train_features].iloc[trn_], train['score'].iloc[trn_]
    8.     val_x, val_y = train[train_features].iloc[val_], train['score'].iloc[val_]
    9.  
    10.     # train_folds = folds[folds['fold'] != fold].reset_index(drop=True)
    11.     # valid_folds = folds[folds['fold'] == fold].reset_index(drop=True)
    12.  
    13.     reg = lgb.LGBMRegressor(**params,n_estimators=1100)
    14.     xgb = XGBRegressor(**xgb_params, n_estimators=1000)
    15.     cat = CatBoostRegressor(iterations=1000,learning_rate=0.03,
    16.                             depth=10,
    17.                             eval_metric='RMSE',
    18.                             random_seed = 42,
    19.                             bagging_temperature = 0.2,
    20.                             od_type='Iter',
    21.                             metric_period = 50,
    22.                             od_wait=20)
    23.     print("-"* 20 + "LightGBM Training" + "-"* 20)
    24.     reg.fit(trn_x, np.log1p(trn_y),eval_set=[(val_x, np.log1p(val_y))],early_stopping_rounds=50,verbose=100,eval_metric='rmse')
    25.     print("-"* 20 + "XGboost Training" + "-"* 20)
    26.     xgb.fit(trn_x, np.log1p(trn_y),eval_set=[(val_x, np.log1p(val_y))],early_stopping_rounds=50,eval_metric='rmse',verbose=100)
    27.     print("-"* 20 + "Catboost Training" + "-"* 20)
    28.     cat.fit(trn_x, np.log1p(trn_y), eval_set=[(val_x, np.log1p(val_y))],early_stopping_rounds=50,use_best_model=True,verbose=100)
    29.     
    30.     imp_df = pd.DataFrame()
    31.     imp_df['feature'] = train_features
    32.     imp_df['gain_reg'] = reg.booster_.feature_importance(importance_type='gain')
    33.     imp_df['fold'] = fold_ + 1
    34.     importances = pd.concat([importances, imp_df], axis=0, sort=False)
    35.     
    36.     
    37.     for model, values in model_infos.items():
    38.         test[model] = values[2][fold_]
    39.         
    40.     for model, values in uspppm_model_infos.items():
    41.         test[f"uspppm_{model}"] = values[2][fold_]
    42.  
    43.         
    44.         
    45.         
    46. #     for f in tqdm(amount_feas, desc="amount_feas 基本聚合特征"):
    47. #         for cate in category_fea:
    48. #             if f != cate:
    49. #                 test['{}_{}_medi'.format(cate, f)] = test.groupby(cate)[f].transform('median')
    50. #                 test['{}_{}_mean'.format(cate, f)] = test.groupby(cate)[f].transform('mean')
    51. #                 test['{}_{}_max'.format(cate, f)] = test.groupby(cate)[f].transform('max')
    52. #                 test['{}_{}_min'.format(cate, f)] = test.groupby(cate)[f].transform('min')
    53. #                 test['{}_{}_std'.format(cate, f)] = test.groupby(cate)[f].transform('std')
    54.             
    55.             
    56.             
    57.     # LightGBM
    58.     oof_reg_preds[val_] = reg.predict(val_x, num_iteration=reg.best_iteration_)
    59. #     oof_reg_preds[oof_reg_preds < 0] = 0
    60.     lgb_preds = reg.predict(test[train_features], num_iteration=reg.best_iteration_)
    61. #     lgb_preds[lgb_preds < 0] = 0
    62.     
    63.     
    64.     # Xgboost
    65.     oof_reg_preds1[val_] = xgb.predict(val_x)
    66.     oof_reg_preds1[oof_reg_preds1 < 0] = 0
    67.     xgb_preds = xgb.predict(test[train_features])
    68. #     xgb_preds[xgb_preds < 0] = 0
    69.     
    70.     # catboost
    71.     oof_reg_preds2[val_] = cat.predict(val_x)
    72.     oof_reg_preds2[oof_reg_preds2 < 0] = 0
    73.     cat_preds = cat.predict(test[train_features])
    74.     cat_preds[xgb_preds < 0] = 0
    75.         
    76. #     merge all prediction
    77.     merge_pred[val_] = oof_reg_preds[val_] * 0.4 + oof_reg_preds1[val_] * 0.3 +oof_reg_preds2[val_] * 0.3
    78.     
    79. #     sub_reg_preds += np.expm1(_preds) / len(folds)
    80. #     sub_reg_preds += np.expm1(_preds) / len(folds)
    81.  
    82.     sub_preds += (lgb_preds / 5) * 0.6 + (xgb_preds / 5) * 0.2 + (cat_preds / 5) * 0.2 #三个模型五折测试集预测结果
    83.     
    84.     sub_reg_preds+=lgb_preds / 5 # lgb五折测试集预测结果
    85. print("lgb",pearsonr(train['score'], np.expm1(oof_reg_preds))[0]) # lgb
    86. print("xgb",pearsonr(train['score'], np.expm1(oof_reg_preds1))[0]) # xgb
    87. print("cat",pearsonr(train['score'], np.expm1(oof_reg_preds2))[0]) # cat
    88. print("xgb lgb cat",pearsonr(train['score'], np.expm1(merge_pred))[0]) # xgb lgb cat
  • 相关阅读:
    HtmlJavaScript的 getElementBYId和querySelector速度性能对比测试 2210011540
    java基础
    Scanner例题讲解
    纳米体育数据足球数据接口：资料库数据包接口文档API示例②
    HIVE内置函数hash() -- 源码解析
    JavaScript 64 JavaScript 函数 64.4 JavaScript 函数 Call
    Unity WebGL 中文输入解决方案（UGUI、TextMeshPro、UIToolkit）
    杀掉进程但是fastapi程序还在运行
    《编译原理》实验二句法分析器
    Blender点操作
  • 原文地址：https://blog.csdn.net/yanqianglifei/article/details/125424006