• paddle2.3和torch1.8在SentenceBert上的性能对比

    写在前面。。。对比了个寂寞，paddle框架目前有bug——或者我代码有bug。。。torch完胜

    目录

    一、背景介绍

    二、paddle和torch实现SentenceBert

    1、SentenceBert

    2、torch实现

    3、paddle实现

    三、效果对比

    一、背景介绍

            为什么要写这样的一篇对比paddle和torch的博客呢？

            百度的paddle开源的一些模型和权重非常好，为了后续工作能更好的利用百度开源的一些模型权重和方法，因此学习好paddle也是有必要的；另外paddle的生态也是越发完善，对标huggingface的transformer和它的模型下载中心，百度也出品了paddlenlp.transformer和paddlehub，同时paddle框架本身也做的越来越好，越来越成熟，它的API和torch也是大同小异，学习成本也比较低，所有更有必要好好学习一下paddle这个框架。

            习惯使用了torch做深度学习，再使用paddle必然也会有一点点不顺手，毕竟API还是有一些差异的，但是这个真的要比torch和tensorflow的差距要小很多。为了能顺利的使用paddle来完成一个文本分类的例子，强迫自己输出一篇博客，把实现SentenceBert文本分类过程中一些问题和经验积累下来，也是作为学习paddle的一个开始。

            总体来说，百度出品的paddle可以用起来了，做的越来越好了，开源的一些模型也是有中文领域的，和torch的差距正在变小，好的东西都是要掌握的。

    SentenceBert做文本分类和匹配是很经典的了，到现在我都认为还是比较能打的，效果确实比较好，实现起来比较容易，当然现在的提示学习呀还有苏神的consenBert在效果上确实要比SentenceBert要好，不过这里重点在于使用paddle实现文本分类的任务，以及对比一下torch，它们在效果和效率上的差异。

    二、paddle和torch实现SentenceBert

    1、SentenceBert

    具体的模型细节就不多说了，放个图做个简介吧

     Bert模型共享参数，把v、u和|u-v|拼接起来，得到分类特征；pooling这里也比较有特色，采用mean_pooling并且把padding部分的0都考虑进来了，消除这部分数据对结果的影响。

    2、torch实现

    模型代码sentence_bert.py

    1. import torch.nn as nn
    2. from transformers import BertModel
    3. from transformers import BertPreTrainedModel
    4. import torch
    5. class SentenceBert(BertPreTrainedModel):
    6.     def __init__(self,config):
    7.         super(SentenceBert,self).__init__(config)
    8.         self.bert = BertModel(config=config)
    9.         self.classifier = nn.Linear(3*config.hidden_size,config.num_labels)
    10.  
    11.  
    12.     def forward(self,inputs):
    13.         input_a = inputs[0]
    14.         input_b = inputs[1]
    15.         output_a = self.bert(**input_a,return_dict=True, output_hidden_states=True)
    16.         output_b = self.bert(**input_b,return_dict=True, output_hidden_states=True)
    17.         #采用最后一层
    18.         embedding_a = output_a.hidden_states[-1]
    19.         embedding_b = output_b.hidden_states[-1]
    20.         embedding_a = self.pooling(embedding_a,input_a)
    21.         embedding_b = self.pooling(embedding_b, input_b)
    22.  
    23.         embedding_abs = torch.abs(embedding_a - embedding_b)
    24.         vectors_concat = []
    25.         vectors_concat.append(embedding_a)
    26.         vectors_concat.append(embedding_b)
    27.         vectors_concat.append(embedding_abs)
    28.         # 列拼接3个768————>3*768
    29.         features = torch.cat(vectors_concat, 1)
    30.         output = self.classifier(features)
    31.         return output
    32.  
    33.  
    34.  
    35.     def pooling(self,token_embeddings,input):
    36.         output_vectors = []
    37.         #attention_mask
    38.         attention_mask = input['attention_mask']
    39.         #[B,L]------>[B,L,1]------>[B,L,768],矩阵的值是0或者1
    40.         input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
    41.         #这里做矩阵点积，就是对元素相乘(序列中padding字符，通过乘以0给去掉了)[B,L,768]
    42.         t = token_embeddings * input_mask_expanded
    43.         #[B,768]
    44.         sum_embeddings = torch.sum(t, 1)
    45.  
    46.         # [B,768],最大值为seq_len
    47.         sum_mask = input_mask_expanded.sum(1)
    48.         #限定每个元素的最小值是1e-9，保证分母不为0
    49.         sum_mask = torch.clamp(sum_mask, min=1e-9)
    50.         #得到最后的具体embedding的每一个维度的值——元素相除
    51.         output_vectors.append(sum_embeddings / sum_mask)
    52.  
    53.         #列拼接
    54.         output_vector = torch.cat(output_vectors, 1)
    55.  
    56.         return  output_vector

    数据读取dataReader_tsv.py

    1.  
    2. from tqdm import tqdm
    3. import torch
    4. import pandas as pd
    5.  
    6. class DataReader(object):
    7.     def __init__(self,tokenizer,filepath,max_len):
    8.         self.tokenizer = tokenizer
    9.         self.filepath = filepath
    10.         self.max_len = max_len
    11.         self.dataList = self.datas_to_torachTensor()
    12.         self.allLength = len(self.dataList)
    13.  
    14.     def convert_text2ids(self,text):
    15.         text = text[0:self.max_len-2]
    16.         inputs = self.tokenizer(text)
    17.  
    18.         input_ids = inputs['input_ids']
    19.         attention_mask = inputs['attention_mask']
    20.         paddings = [0] * (self.max_len - len(input_ids))
    21.         input_ids += paddings
    22.         attention_mask += paddings
    23.  
    24.         token_type_id = [0] * self.max_len
    25.  
    26.         return input_ids, attention_mask, token_type_id
    27.  
    28.  
    29.     def datas_to_torachTensor(self):
    30.         # with open(self.filepath,'r',encoding='utf-8') as f:
    31.         #     lines = f.readlines()
    32.  
    33.         df = pd.read_csv(self.filepath,sep='\t')
    34.         df.dropna(inplace=True)
    35.         lines_a = df['sentence1'].values.tolist()
    36.         lines_b = df['sentence2'].values.tolist()
    37.         labels = df['label'].values.tolist()
    38.  
    39.  
    40.         res = []
    41.         for line_a,line_b,label in tqdm(zip(lines_a,lines_b,labels),desc='tokenization',ncols=50):
    42.             temp = []
    43.             try:
    44.                 if line_a != '' and len(line_a) > 0 and line_b != '' and len(line_b) > 0:
    45.                     input_ids_a, attention_mask_a, token_type_id_a = self.convert_text2ids(text=line_a)
    46.                     input_ids_a = torch.as_tensor(input_ids_a, dtype=torch.long)
    47.                     attention_mask_a = torch.as_tensor(attention_mask_a, dtype=torch.long)
    48.                     token_type_id_a = torch.as_tensor(token_type_id_a, dtype=torch.long)
    49.                     temp.append(input_ids_a)
    50.                     temp.append(attention_mask_a)
    51.                     temp.append(token_type_id_a)
    52.  
    53.                     input_ids_b, attention_mask_b, token_type_id_b = self.convert_text2ids(text=line_b)
    54.                     input_ids_b = torch.as_tensor(input_ids_b, dtype=torch.long)
    55.                     attention_mask_b = torch.as_tensor(attention_mask_b, dtype=torch.long)
    56.                     token_type_id_b = torch.as_tensor(token_type_id_b, dtype=torch.long)
    57.                     temp.append(input_ids_b)
    58.                     temp.append(attention_mask_b)
    59.                     temp.append(token_type_id_b)
    60.  
    61.                     
    62.                     label = torch.as_tensor(label,dtype=torch.long)
    63.                    
    64.                     temp.append(label)
    65.  
    66.                     res.append(temp)
    67.             except Exception as e:
    68.                 print(e)
    69.         return res
    70.  
    71.     def __getitem__(self, item):
    72.         input_ids_a = self.dataList[item][0]
    73.         attention_mask_a = self.dataList[item][1]
    74.         token_type_id_a = self.dataList[item][2]
    75.  
    76.         input_ids_b = self.dataList[item][3]
    77.         attention_mask_b = self.dataList[item][4]
    78.         token_type_id_b = self.dataList[item][5]
    79.  
    80.         label = self.dataList[item][6]
    81.         
    82.         return input_ids_a, attention_mask_a, token_type_id_a, input_ids_b, attention_mask_b, token_type_id_b, label
    83.  
    84.  
    85.     def __len__(self):
    86.         return self.allLength

    模型训练train_sentence_bert.py

    1. import torch
    2. import argparse
    3. from data_reader.dataReader_tsv import DataReader
    4. from model.sentence_bert_copy import SentenceBert
    5. from torch.utils.data import DataLoader
    6. import torch.nn.functional as F
    7. from torch.optim import AdamW
    8. from torch.optim.lr_scheduler import ReduceLROnPlateau
    9. from transformers import BertTokenizer,BertConfig
    10. import os
    11. from tools.progressbar import ProgressBar
    12. from tools.log import Logger
    13. from datetime import datetime
    14. from torch.nn.utils.rnn import pad_sequence
    15. logger = Logger('sbert_loger',log_level=10,log_file = "./log_output/sbert_pawsx_torch.log").logger
    16. os.environ['CUDA_VISIBLE_DEVICES'] = "0"
    17.  
    18. def parse_args():
    19.     parser = argparse.ArgumentParser(formatter_class=argparse.ArgumentDefaultsHelpFormatter)
    20.     parser.add_argument("--max_len",type=int,default=64)
    21.     parser.add_argument("--train_file", type=str,default='./data/paws_x/translated_train.tsv', help="train text file")
    22.     parser.add_argument("--val_file", type=str, default='./data/paws_x/dev_2k.tsv',help="val text file")
    23.     parser.add_argument("--pretrained", type=str, default="./pretrained_models/torch/chinese-bert-wwm-ext", help="huggingface pretrained model")
    24.     parser.add_argument("--model_out", type=str, default="./output/pawsx", help="model output path")
    25.     parser.add_argument("--batch_size", type=int, default=128, help="batch size")
    26.     parser.add_argument("--epochs", type=int, default=10, help="epochs")
    27.     parser.add_argument("--lr", type=float, default=1e-5, help="epochs")
    28.     parser.add_argument("--task_type",type=str,default='classification')
    29.     args = parser.parse_args()
    30.     return args
    31.  
    32. def collate_fn(batch):
    33.     input_ids_a, attention_mask_a, token_type_id_a, input_ids_b, attention_mask_b, token_type_id_b, label = zip(*batch)
    34.     input_ids_a = pad_sequence(input_ids_a,batch_first=True,padding_value=0)
    35.     attention_mask_a = pad_sequence(attention_mask_a, batch_first=True, padding_value=0)
    36.     token_type_id_a = pad_sequence(token_type_id_a, batch_first=True, padding_value=0)
    37.     input_ids_b = pad_sequence(input_ids_b, batch_first=True, padding_value=0)
    38.     attention_mask_b = attention_mask_b(attention_mask_b, batch_first=True, padding_value=0)
    39.     token_type_id_b = pad_sequence(token_type_id_b, batch_first=True, padding_value=0)
    40.     label = torch.stack(label)
    41.     return input_ids_a, attention_mask_a, token_type_id_a, input_ids_b, attention_mask_b, token_type_id_b, label
    42.  
    43. def train(args):
    44.     logger.info("args: %s",args)
    45.     tokenizer = BertTokenizer.from_pretrained(args.pretrained)
    46.     config = BertConfig.from_pretrained(args.pretrained)
    47.     device = "cuda" if torch.cuda.is_available() else "cpu"
    48.     task_type = args.task_type
    49.     model = SentenceBert.from_pretrained(config=config, pretrained_model_name_or_path=args.pretrained)
    50.     model.to(device)
    51.  
    52.     train_dataset = DataReader(tokenizer=tokenizer,filepath=args.train_file,max_len=args.max_len)
    53.     train_dataloader = DataLoader(train_dataset,batch_size=args.batch_size,shuffle=True, collate_fn = collate_fn)
    54.  
    55.     val_dataset = DataReader(tokenizer=tokenizer,filepath=args.val_file,max_len=args.max_len)
    56.     val_dataloader = DataLoader(val_dataset, batch_size=args.batch_size, shuffle=False,collate_fn=collate_fn)
    57.  
    58.     optimizer = AdamW(model.parameters(),lr=args.lr)
    59.     scheduler = ReduceLROnPlateau(optimizer=optimizer,mode='max',factor=0.5, patience=5)
    60.  
    61.     model.train()
    62.     logger.info("***** Running training *****")
    63.     logger.info("  Num examples = %d", len(train_dataloader))
    64.     logger.info("  Num Epochs = %d", args.epochs)
    65.  
    66.     time_srt = datetime.now().strftime('%Y-%m-%d')
    67.     save_path = os.path.join(args.model_out, "paddle_" + time_srt)
    68.  
    69.     patience = 5
    70.     no_best_epoch = 0
    71.     best_acc = 0.0
    72.  
    73.     for epoch in range(args.epochs):
    74.         pbar = ProgressBar(n_total=len(train_dataloader), desc='Training')
    75.         for step,batch in enumerate(train_dataloader):
    76.             batch = [t.to(device) for t in batch]
    77.             inputs_a = {'input_ids':batch[0],'attention_mask':batch[1],'token_type_ids':batch[2]}
    78.             inputs_b = {'input_ids': batch[3], 'attention_mask': batch[4], 'token_type_ids': batch[5]}
    79.             labels = batch[6]
    80.             inputs = []
    81.             inputs.append(inputs_a)
    82.             inputs.append(inputs_b)
    83.             output = model(inputs)
    84.             loss = F.cross_entropy(output,labels)
    85.  
    86.             loss.backward()
    87.             optimizer.step()
    88.             optimizer.zero_grad()
    89.             pbar(step, {'loss':loss.item()})
    90.  
    91.         val_acc = valdation(model, val_dataloader, device, task_type)
    92.         scheduler.step(val_acc)
    93.  
    94.         if val_acc > best_acc:
    95.             best_acc = val_acc
    96.             if not os.path.exists(save_path):
    97.                 os.makedirs(save_path)
    98.             logger.info("save model")
    99.             model.save_pretrained(save_path)
    100.             tokenizer.save_vocabulary(save_path)
    101.             no_best_epoch = 0
    102.         else:
    103.             no_best_epoch += 1
    104.         logger.info("val_acc:%.4f------best_acc:%.4f" % (val_acc, best_acc))
    105.  
    106.         if no_best_epoch >= patience:
    107.             logger.info("taining finished because of no improving" )
    108.             break
    109.  
    110. def valdation(model,val_dataloader,device):
    111.     total = 0
    112.     total_correct = 0
    113.     model.eval()
    114.     with torch.no_grad():
    115.         pbar = ProgressBar(n_total=len(val_dataloader), desc='evaldation')
    116.         for step, batch in enumerate(val_dataloader):
    117.             batch = [t.to(device) for t in batch]
    118.             inputs_a = {'input_ids': batch[0], 'attention_mask': batch[1], 'token_type_ids': batch[2]}
    119.             inputs_b = {'input_ids': batch[3], 'attention_mask': batch[4], 'token_type_ids': batch[5]}
    120.             labels = batch[6]
    121.             inputs = []
    122.             inputs.append(inputs_a)
    123.             inputs.append(inputs_b)
    124.             output = model(inputs)
    125.             pred = torch.argmax(output, dim=1)
    126.             correct = (labels == pred).sum()
    127.             total_correct += correct
    128.             total += labels.size()[0]
    129.             loss = F.cross_entropy(output, labels)
    130.             pbar(step, {'loss': loss.item()})
    131.  
    132.     acc = total_correct / total
    133.     return acc
    134.  
    135. def main():
    136.     torch.manual_seed(1)
    137.     torch.cuda.manual_seed(1)
    138.     args =parse_args()
    139.     train(args)
    140.  
    141. if __name__ == '__main__':
    142.     main()

    torch的代码比较简单，就不做多的解释和说明了，有一个需要注意的地方就是记得要设置随机种子。

    torch.manual_seed(1)
torch.cuda.manual_seed(1)

    下面来看paddle的实现

    3、paddle实现

    模型组网代码

    模型实现，模型计算逻辑使用的API和torch基本相同，只需要修改如下：

    torch.cat()------------------paddle.concat()

    torch.clamp()------------------paddle.clip()

    ......

    其他的基本只需要把torch修改为paddle就可以了，主要的不同用法：

    transformer中的bert输出是一个字典

    bert(**input_a,return_dict=True, output_hidden_states=True).hidden_states[-1]

    paddlenlp.transformer中的bert输出是一个list 第一个元素是每一层的hidden_states;第二个元素就是pooled_output也就是cls向量；所以一般采用如下代码：
    bert(**input_a, output_hidden_states=True)[0][-1]最后一层的hidden_states

    关于模型权重和配置文件的加载也不一样

    paddle在指定模型名称的时候会采取自动下载模型和权重，然后再加载的方式

    SentenceBert.from_pretrained('bert-wwm-ext-chinese')

    会把模型权重和配置文件保存在/root/.paddlenlp/models/bert-wwm-ext-chinese路径下：

     内容如上图所示

    如果要像torch一样加载本地的bert等预训练模型，需要先使用paddlenlp.transformers把模型权重下载后，然后使用model.save_pretrained(local_path)保存模型权重和模型配置文件(自动下载的时候是没有模型配置文件的)，tokenizer.save_pretrained(local_path)保存词表。上述保存的文件名和自动下载的是不一样的：

     注意模型权重文件model_state.pdparams；模型配置文件model_config.json，并且model_config.json和torch下的bert模型配置文件格式不同：

    {
        "init_args":[
            {
                "vocab_size":21128,
                "hidden_size":768,
                "num_hidden_layers":12,
                "num_attention_heads":12,
                "intermediate_size":3072,
                "hidden_act":"gelu",
                "hidden_dropout_prob":0.1,
                "attention_probs_dropout_prob":0.1,
                "max_position_embeddings":512,
                "type_vocab_size":2,
                "initializer_range":0.02,
                "pad_token_id":0,
                "init_class":"BertModel"
            }
        ],
        "init_class":"SentenceBert"
    }

    模型组网如下sentence_bert.py：

    1. from paddlenlp.transformers import BertPretrainedModel,BertModel
    2. import paddle.nn as nn
    3. import paddle
    4.  
    5.  
    6. class SentenceBert(BertPretrainedModel):
    7.     """
    8.     __init__(self,bert) 一定要带一个参数，外层from_pretrained()才能成功
    9.     bert会在from_pretrained()的时候初始化成功然后传入
    10.     """
    11.     # base_model_prefix = "bert"
    12.     def __init__(self, bert):
    13.         super(SentenceBert,self).__init__()
    14.         self.bert = bert
    15.         self.classifier = nn.Linear(3*768,2)
    16.  
    17.     def forward(self, inputs, **kwargs):
    18.         input_a = inputs[0]
    19.         input_b = inputs[1]
    20.         output_a = self.bert(**input_a, output_hidden_states=True)
    21.         output_b = self.bert(**input_b, output_hidden_states=True)
    22.         # 采用最后一层
    23.         embedding_a = output_a[0][-1]
    24.         embedding_b = output_b[0][-1]
    25.         embedding_a = self.pooling(embedding_a, input_a)
    26.         embedding_b = self.pooling(embedding_b, input_b)
    27.  
    28.         embedding_abs = paddle.abs(embedding_a - embedding_b)
    29.         vectors_concat = []
    30.         vectors_concat.append(embedding_a)
    31.         vectors_concat.append(embedding_b)
    32.         vectors_concat.append(embedding_abs)
    33.         # 列拼接3个768————>3*768
    34.         features = paddle.concat(vectors_concat,axis=-1)
    35.         output = self.classifier(features)
    36.         return output
    37.  
    38.     def pooling(self, token_embeddings, input):
    39.         output_vectors = []
    40.         # attention_mask
    41.         attention_mask = input['attention_mask']
    42.         # [B,L]------>[B,L,1]------>[B,L,768],矩阵的值是0或者1
    43.         # input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
    44.         input_mask_expanded = paddle.unsqueeze(attention_mask,-1)
    45.         input_mask_expanded = paddle.expand(input_mask_expanded,shape=[input_mask_expanded.shape[0],input_mask_expanded.shape[1],token_embeddings.shape[-1]])
    46.  
    47.         # 这里做矩阵点积，就是对元素相乘(序列中padding字符，通过乘以0给去掉了)[B,L,768]
    48.         t = token_embeddings * input_mask_expanded
    49.  
    50.         # [B,768]
    51.         sum_embeddings = paddle.sum(t, 1)
    52.  
    53.         # [B,768],最大值为seq_len
    54.         sum_mask = input_mask_expanded.sum(1)
    55.         # 限定每个元素的最小值是1e-9，保证分母不为0
    56.         sum_mask = paddle.clip(sum_mask, min=1e-9)
    57.         # 得到最后的具体embedding的每一个维度的值——元素相除
    58.         output_vectors.append(sum_embeddings / sum_mask)
    59.  
    60.         # 列拼接
    61.         output_vector = paddle.concat(output_vectors, 1)
    62.  
    63.         return output_vector

    上述组网的方式，加载权重的时候要注意这个init中的bert不是我们传入的参数，而是

    SentenceBert.from_pretrained(pretrained_model_name_or_path=args.pretrained)自动传入的参数

    但是在模型结构定义的时候一定的传入一个参数才不会报错。

    数据加载dataReader.py

    1. from paddle.io import Dataset
    2. import pandas as pd
    3. from tqdm import tqdm
    4. import paddle
    5.  
    6. class DataReader(Dataset):
    7.     def __init__(self, tokenizer, filepath, max_len):
    8.         self.tokenizer = tokenizer
    9.         self.filepath = filepath
    10.         self.max_len = max_len
    11.         self.dataList = self.datas_to_torachTensor()
    12.         self.allLength = len(self.dataList)
    13.  
    14.     def convert_text2ids(self, text):
    15.         text = text[0:self.max_len - 2]
    16.         inputs = self.tokenizer(text)
    17.  
    18.         input_ids = inputs['input_ids']
    19.         attention_mask = [1]*len(input_ids)
    20.         token_type_id = inputs["token_type_ids"]
    21.  
    22.         return input_ids, attention_mask, token_type_id
    23.  
    24.     def datas_to_torachTensor(self):
    25.         df = pd.read_csv(self.filepath, sep='\t')
    26.         df.dropna(inplace=True)
    27.         lines_a = df['sentence1'].values.tolist()
    28.         lines_b = df['sentence2'].values.tolist()
    29.         labels = df['label'].values.tolist()
    30.  
    31.         res = []
    32.         for line_a, line_b, label in tqdm(zip(lines_a[0:], lines_b[0:], labels[0:]), desc='tokenization', ncols=50):
    33.             temp = []
    34.             try:
    35.                 if line_a != '' and len(line_a) > 0 and line_b != '' and len(line_b) > 0:
    36.                     input_ids_a, attention_mask_a, token_type_id_a = self.convert_text2ids(text=line_a)
    37.                     input_ids_a = paddle.to_tensor(input_ids_a,dtype='int64')
    38.                     attention_mask_a = paddle.to_tensor(attention_mask_a, dtype='int64')
    39.                     token_type_id_a = paddle.to_tensor(token_type_id_a, dtype='int64')
    40.                     temp.append(input_ids_a)
    41.                     temp.append(attention_mask_a)
    42.                     temp.append(token_type_id_a)
    43.  
    44.                     input_ids_b, attention_mask_b, token_type_id_b = self.convert_text2ids(text=line_b)
    45.                     input_ids_b = paddle.to_tensor(input_ids_b, dtype='int64')
    46.                     attention_mask_b = paddle.to_tensor(attention_mask_b, dtype='int64')
    47.                     token_type_id_b = paddle.to_tensor(token_type_id_b, dtype='int64')
    48.                     temp.append(input_ids_b)
    49.                     temp.append(attention_mask_b)
    50.                     temp.append(token_type_id_b)
    51.                     label = paddle.to_tensor(label, dtype='int64')
    52.                     temp.append(label)
    53.                     res.append(temp)
    54.             except Exception as e:
    55.                 print(e)
    56.         return res
    57.  
    58.     def __getitem__(self, item):
    59.         input_ids_a = self.dataList[item][0]
    60.         attention_mask_a = self.dataList[item][1]
    61.         token_type_id_a = self.dataList[item][2]
    62.  
    63.         input_ids_b = self.dataList[item][3]
    64.         attention_mask_b = self.dataList[item][4]
    65.         token_type_id_b = self.dataList[item][5]
    66.  
    67.         label = self.dataList[item][6]
    68.  
    69.         return input_ids_a, attention_mask_a, token_type_id_a, input_ids_b, attention_mask_b, token_type_id_b, label
    70.  
    71.     def __len__(self):
    72.         return self.allLength

    模型训练train_sentence_bert.py

    注意的是这里是随机种子的设置比较严格仅仅一个paddle.seed()并不能保证训练的可复现和一致。还需要其他的设置：

    paddle.seed(1)
random.seed(1)
np.random.seed(1)

    这样设置比较靠谱，有可能不知道的地方隐式的调用了上面的random和numpy库，保险起见最好设置一下比较好。paddle的

    DataLoader(train_dataset,batch_size=args.batch_size,shuffle=True, collate_fn=collate_fn)

    这句代码中的 shuffle 应该是引用了random包 所以每次batch出来的数据不一样，所以random也要设置seed

    1. from data_reader.dataReader import DataReader
    2. from model.sentence_bert import SentenceBert
    3. import argparse
    4. from tools.progressbar import ProgressBar
    5. from tools.log import Logger
    6. from datetime import datetime
    7. import os
    8. from paddlenlp.transformers import BertTokenizer
    9. from paddle.io import DataLoader
    10. from paddle.optimizer import AdamW
    11. from paddle.optimizer.lr import ReduceOnPlateau
    12. import paddle
    13. import paddle.nn.functional  as F
    14. import numpy as np
    15. import random
    16.  
    17. paddle.seed(1)
    18. random.seed(1)
    19. np.random.seed(1)
    20.  
    21. logger = Logger('paddle_sbert_loger',log_level=10,log_file='./log_output/pawsx_sbert_paddle.log').logger
    22. paddle.set_device('gpu:0')
    23.  
    24. def parse_args():
    25.     parser = argparse.ArgumentParser(formatter_class=argparse.ArgumentDefaultsHelpFormatter)
    26.     parser.add_argument("--max_len",type=int,default=64)
    27.     parser.add_argument("--train_file", type=str,default='./data/paws_x/translated_train.tsv', help="train text file")
    28.     parser.add_argument("--val_file", type=str, default='./data/paws_x/dev_2k.tsv',help="val text file")
    29.     parser.add_argument("--pretrained", type=str, default="pretrained_models/paddle/bert-wwm-ext-chinese", help="huggingface pretrained model")
    30.     parser.add_argument("--model_out", type=str, default="./output", help="model output path")
    31.     parser.add_argument("--batch_size", type=int, default=64, help="batch size")
    32.     parser.add_argument("--epochs", type=int, default=10, help="epochs")
    33.     parser.add_argument("--lr", type=float, default=1e-5, help="epochs")
    34.     parser.add_argument("--task_type",type=str,default='classification')
    35.     args = parser.parse_args()
    36.     return args
    37.  
    38. def collate_fn(batch):
    39.     input_ids_a, attention_mask_a, token_type_id_a, input_ids_b, attention_mask_b, token_type_id_b, label = zip(*batch)
    40.     print(input_ids_a)
    41.     exit()
    42.     bz = len(input_ids_a)
    43.     max_a = 0
    44.     for i in range(len(input_ids_a)):
    45.         if input_ids_a[i].shape[0] > max_a:
    46.             max_a = input_ids_a[i].shape[0]
    47.  
    48.     input_ids_a_pad = paddle.zeros(shape=[bz,max_a],dtype="int64")
    49.     for i in range(len(input_ids_a)):
    50.         l = input_ids_a[i].shape[0]
    51.         input_ids_a_pad[i,0:l] = input_ids_a[i]
    52.  
    53.     attention_mask_a_pad = paddle.zeros(shape=[bz, max_a],dtype="int64")
    54.     for i in range(len(input_ids_a)):
    55.         l = attention_mask_a[i].shape[0]
    56.         attention_mask_a_pad[i, 0:l] = attention_mask_a[i]
    57.  
    58.     token_type_id_a_pad = paddle.zeros(shape=[bz, max_a],dtype="int64")
    59.     for i in range(len(input_ids_a)):
    60.         l = token_type_id_a[i].shape[0]
    61.         token_type_id_a_pad[i, 0:l] = token_type_id_a[i]
    62.  
    63.     max_b = 0
    64.     for i in range(len(input_ids_b)):
    65.         if input_ids_b[i].shape[0] > max_b:
    66.             max_b = input_ids_b[i].shape[0]
    67.  
    68.     input_ids_b_pad = paddle.zeros(shape=[bz, max_b],dtype="int64")
    69.     for i in range(len(input_ids_b)):
    70.         l = input_ids_b[i].shape[0]
    71.         input_ids_b_pad[i, 0:l] = input_ids_b[i]
    72.  
    73.     attention_mask_b_pad = paddle.zeros(shape=[bz, max_b],dtype="int64")
    74.     for i in range(len(input_ids_b)):
    75.         l = attention_mask_b[i].shape[0]
    76.         attention_mask_b_pad[i, 0:l] = attention_mask_b[i]
    77.  
    78.     token_type_id_b_pad = paddle.zeros(shape=[bz, max_b],dtype="int64")
    79.     for i in range(len(input_ids_b)):
    80.         l = token_type_id_b[i].shape[0]
    81.         token_type_id_b_pad[i, 0:l] = token_type_id_b[i]
    82.  
    83.     label = paddle.stack(label,axis=0)
    84.     return input_ids_a_pad, attention_mask_a_pad, token_type_id_a_pad, input_ids_b_pad, attention_mask_b_pad, token_type_id_b_pad, label
    85.  
    86. def train(args):
    87.     logger.info("args: %s",args)
    88.     ##会自动下载相应的权重和配置文件
    89.     #tokenizer = BertTokenizer.from_pretrained('bert-wwm-ext-chinese')
    90.     #model = SentenceBert.from_pretrained('bert-wwm-ext-chinese')
    91.  
    92.     tokenizer = BertTokenizer.from_pretrained(pretrained_model_name_or_path=args.pretrained)
    93.     model = SentenceBert.from_pretrained(pretrained_model_name_or_path=args.pretrained)
    94.  
    95.     train_dataset = DataReader(tokenizer=tokenizer,filepath=args.train_file,max_len=args.max_len)
    96.     train_dataloader = DataLoader(train_dataset,batch_size=args.batch_size,shuffle=True, collate_fn=collate_fn)
    97.  
    98.     val_dataset = DataReader(tokenizer=tokenizer,filepath=args.val_file,max_len=args.max_len)
    99.     val_dataloader = DataLoader(val_dataset, batch_size=args.batch_size, shuffle=False, collate_fn=collate_fn)
    100.  
    101.     optimizer = AdamW(parameters = model.parameters(),learning_rate = args.lr)
    102.     scheduler = ReduceOnPlateau(learning_rate= args.lr,mode='max',factor=0.5, patience=5)
    103.     model.train()
    104.     logger.info("***** Running training *****")
    105.     logger.info("  Num examples = %d", len(train_dataloader))
    106.     logger.info("  Num Epochs = %d", args.epochs)
    107.  
    108.     time_srt = datetime.now().strftime('%Y-%m-%d')
    109.     save_path = os.path.join(args.model_out,"paddle_" + time_srt)
    110.  
    111.     best_acc = 0.0
    112.     patience = 5
    113.     no_best_epoch = 0
    114.     model.train()
    115.     for epoch in range(args.epochs):
    116.         pbar = ProgressBar(n_total=len(train_dataloader), desc='Training')
    117.         for step,batch in enumerate(train_dataloader):
    118.             inputs_a = {'input_ids':batch[0],'attention_mask':batch[1],'token_type_ids':batch[2]}
    119.             inputs_b = {'input_ids': batch[3], 'attention_mask': batch[4], 'token_type_ids': batch[5]}
    120.             labels = batch[6]
    121.             inputs = []
    122.             inputs.append(inputs_a)
    123.             inputs.append(inputs_b)
    124.             output = model(inputs)
    125.             loss = F.cross_entropy(output,labels)
    126.             loss.backward()
    127.             optimizer.step()
    128.             optimizer.clear_grad()
    129.             pbar(step, {'loss':loss.item()})
    130.  
    131.         val_acc = valdation(model, val_dataloader)
    132.         scheduler.step(val_acc)
    133.  
    134.         if val_acc > best_acc:
    135.             best_acc = val_acc
    136.  
    137.             if not os.path.exists(save_path):
    138.                 os.makedirs(save_path)
    139.             logger.info("save model")
    140.             model.save_pretrained(save_path)
    141.             tokenizer.save_pretrained(save_path)
    142.             no_best_epoch = 0
    143.         else:
    144.             no_best_epoch += 1
    145.         logger.info("val_acc:%.4f------best_acc:%.4f" % ( val_acc, best_acc))
    146.  
    147.         if no_best_epoch >= patience:
    148.             logger.info("taining finished because of no improving" )
    149.             exit()
    150.  
    151.     
    152. def valdation(model,val_dataloader):
    153.     total = 0
    154.     total_correct = 0
    155.     model.eval()
    156.     with paddle.no_grad():
    157.         pbar = ProgressBar(n_total=len(val_dataloader), desc='evaldation')
    158.         for step, batch in enumerate(val_dataloader):
    159.             inputs_a = {'input_ids': batch[0], 'attention_mask': batch[1], 'token_type_ids': batch[2]}
    160.             inputs_b = {'input_ids': batch[3], 'attention_mask': batch[4], 'token_type_ids': batch[5]}
    161.             labels = paddle.squeeze(batch[6])
    162.             inputs = []
    163.             inputs.append(inputs_a)
    164.             inputs.append(inputs_b)
    165.             output = model(inputs)
    166.             pred = paddle.argmax(output, axis=1)
    167.             correct = (labels == pred).sum()
    168.             total_correct += correct
    169.             total += labels.shape[0]
    170.             loss = F.cross_entropy(output, labels)
    171.             pbar(step, {'loss': loss.item()})
    172.  
    173.     acc = total_correct / total
    174.     return acc
    175.  
    176. def main():
    177.     args =parse_args()
    178.     train(args)
    179.  
    180. if __name__ == '__main__':
    181.     main()

    三、效果对比

    直接看看模型训练的时间和准确率(更科学一点的应该是采用F1值召回率等等)

    torch结果

    torch的训练时间和准确率如下图

     paddle目前框架有问题，设置随机种子不生效，导致结果不可复现，每次训练结果乱飞(0.65-0.68之间)，尝试如下设置后，均没有效果

    paddle.seed(100)
random.seed(100)
np.random.seed(100)
    FLAGS_cudnn_deterministic = True python train....

     需要等官方提供解决办法，看来paddle还是有很多bug呀，白开心一场呀。。。至于后续的对比就没有啥意义了；后面再来修改这篇博客吧

    和paddle官方沟通情况

  • 相关阅读:
    【Java8】对JSONArray 按指定字段排序，升序和倒序，经典
    React组件进阶
    【前端实习生备战秋招】—计算机网络面试题汇总，建议收藏系列
    钉钉添加自定义机器人，实现每周定时@某人
    C# 内存泄漏之 Internal 关键词代表什么？
    .Net Web项目创建比较不错的参考文章
    分布式ID生成器-rain
    python 自动化部署SpringBoot到远程linux系统
    Vue 中利用 template标签遍历多维数组
    docker 存储挂载比较
  • 原文地址：https://blog.csdn.net/HUSTHY/article/details/126144004