• 第86步 时间序列建模实战：Transformer回归建模

    基于WIN10的64位系统演示

    一、写在前面

    这一期，我们介绍Transformer回归。

    同样，这里使用这个数据：

    《PLoS One》2015年一篇题目为《Comparison of Two Hybrid Models for Forecasting the Incidence of Hemorrhagic Fever with Renal Syndrome in Jiangsu Province, China》文章的公开数据做演示。数据为江苏省2004年1月至2012年12月肾综合症出血热月发病率。运用2004年1月至2011年12月的数据预测2012年12个月的发病率数据。

    二、Transformer回归

    （1）原理

    Transformer框架原本是为NLP任务，特别是机器翻译而设计的。但由于其独特的自注意力机制，Transformer在处理顺序数据时表现出色，因此被广泛应用于各种序列数据任务，包括回归任务。

    （a）回归任务中的Transformer：

    （a1）在回归任务中，Transformer可以捕捉数据中的长期依赖关系。例如，在时间序列数据中，Transformer可以捕捉时间点之间的关系，即使这些时间点相隔很远。

    （a2）为回归任务使用Transformer时，通常需要稍微调整模型结构，特别是模型的输出部分。原始的Transformer用于生成序列，但在回归任务中，我们通常需要一个单一的实数作为输出。

    （b）Transformer的优点：

    （b1）自注意力机制：可以捕捉序列中的任意位置间的依赖关系，而不像RNN那样依赖于前面的信息。

    （b2）并行计算：与RNN或LSTM不同，Transformer不需要按顺序处理数据，因此更容易并行处理，提高训练速度。

    （b3）可扩展性：可以通过堆叠多个Transformer层来捕捉复杂的模式和关系。

    模型解释性：由于自注意力机制，我们可以可视化哪些输入位置对于特定输出最为重要，这增加了模型的解释性。

    （c）Transformer的缺点：

    （c1）计算需求：尽管可以并行化，但Transformer模型，特别是大型模型，仍然需要大量的计算资源。

    （c2）过拟合：在小型数据集上，特别是没有足够的正则化时，Transformer可能会过拟合。

    （c3）长序列的挑战：尽管Transformer可以处理长序列，但由于自注意力机制的复杂性，处理非常长的序列仍然是一个挑战。为此，研究人员已经提出了许多变种，例如Reformer。

    总体而言，Transformer提供了一个强大的框架来处理各种序列数据任务。

    （2）单步滚动预测

    1. import pandas as pd
    2. import numpy as np
    3. from sklearn.metrics import mean_absolute_error, mean_squared_error
    4. from tensorflow.python.keras.models import Sequential
    5. from tensorflow.python.keras import layers, models, optimizers
    6. from tensorflow.python.keras.optimizers import adam_v2
    7.  
    8. # 读取数据
    9. data = pd.read_csv('data.csv')
    10.  
    11. # 将时间列转换为日期格式
    12. data['time'] = pd.to_datetime(data['time'], format='%b-%y')
    13.  
    14. # 创建滞后期特征
    15. lag_period = 6
    16. for i in range(lag_period, 0, -1):
    17.     data[f'lag_{i}'] = data['incidence'].shift(lag_period - i + 1)
    18.  
    19. # 删除包含 NaN 的行
    20. data = data.dropna().reset_index(drop=True)
    21.  
    22. # 划分训练集和验证集
    23. train_data = data[(data['time'] >= '2004-01-01') & (data['time'] <= '2011-12-31')]
    24. validation_data = data[(data['time'] >= '2012-01-01') & (data['time'] <= '2012-12-31')]
    25.  
    26. # 定义特征和目标变量
    27. X_train = train_data[['lag_1', 'lag_2', 'lag_3', 'lag_4', 'lag_5', 'lag_6']].values
    28. y_train = train_data['incidence'].values
    29. X_validation = validation_data[['lag_1', 'lag_2', 'lag_3', 'lag_4', 'lag_5', 'lag_6']].values
    30. y_validation = validation_data['incidence'].values
    31.  
    32. # 对于Transformer，我们需要将输入数据重塑为 [samples, timesteps, features]
    33. X_train = X_train.reshape(X_train.shape[0], X_train.shape[1], 1)
    34. X_validation = X_validation.reshape(X_validation.shape[0], X_validation.shape[1], 1)
    35.  
    36. # Transformer的一些参数设置
    37. d_model = 128
    38. num_heads = 4
    39.  
    40. # 构建Transformer回归模型
    41. input_layer = layers.Input(shape=(X_train.shape[1], 1))
    42.  
    43. # Linear Embedding
    44. x = layers.Dense(d_model)(input_layer)
    45.  
    46. # Multi Head Self Attention
    47. x = layers.MultiHeadAttention(num_heads=num_heads, key_dim=d_model)(x, x)
    48.  
    49. # Feed Forward Neural Networks
    50. x = layers.GlobalAveragePooling1D()(x)
    51. x = layers.Dropout(0.1)(x)
    52. x = layers.Dense(50, activation='relu')(x)
    53. x = layers.Dropout(0.1)(x)
    54. output_layer = layers.Dense(1)(x)
    55.  
    56. model = models.Model(inputs=input_layer, outputs=output_layer)
    57.  
    58. model.compile(optimizer=adam_v2.Adam(learning_rate=0.001), loss='mse')
    59.  
    60. # 训练模型
    61. history = model.fit(X_train, y_train, epochs=200, batch_size=32, validation_data=(X_validation, y_validation), verbose=0)
    62.  
    63. # 单步滚动预测函数
    64. def rolling_forecast(model, initial_features, n_forecasts):
    65.     forecasts = []
    66.     current_features = initial_features.copy()
    67.  
    68.     for i in range(n_forecasts):
    69.         # 使用当前的特征进行预测
    70.         forecast = model.predict(current_features.reshape(1, len(current_features), 1)).flatten()[0]
    71.         forecasts.append(forecast)
    72.  
    73.         # 更新特征，用新的预测值替换最旧的特征
    74.         current_features = np.roll(current_features, shift=-1)
    75.         current_features[-1] = forecast
    76.  
    77.     return np.array(forecasts)
    78.  
    79. # 使用训练集的最后6个数据点作为初始特征
    80. initial_features = X_train[-1].flatten()
    81.  
    82. # 使用单步滚动预测方法预测验证集
    83. y_validation_pred = rolling_forecast(model, initial_features, len(X_validation))
    84.  
    85. # 计算训练集上的MAE, MAPE, MSE 和 RMSE
    86. mae_train = mean_absolute_error(y_train, model.predict(X_train).flatten())
    87. mape_train = np.mean(np.abs((y_train - model.predict(X_train).flatten()) / y_train))
    88. mse_train = mean_squared_error(y_train, model.predict(X_train).flatten())
    89. rmse_train = np.sqrt(mse_train)
    90.  
    91. # 计算验证集上的MAE, MAPE, MSE 和 RMSE
    92. mae_validation = mean_absolute_error(y_validation, y_validation_pred)
    93. mape_validation = np.mean(np.abs((y_validation - y_validation_pred) / y_validation))
    94. mse_validation = mean_squared_error(y_validation, y_validation_pred)
    95. rmse_validation = np.sqrt(mse_validation)
    96.  
    97. print("验证集：", mae_validation, mape_validation, mse_validation, rmse_validation)
    98. print("训练集：", mae_train, mape_train, mse_train, rmse_train)

    看结果：

    （3）多步滚动预测-vol. 1

    1. import pandas as pd
    2. import numpy as np
    3. from sklearn.metrics import mean_absolute_error, mean_squared_error
    4. import tensorflow as tf
    5. from tensorflow.python.keras.models import Model
    6. from tensorflow.python.keras.layers import Input, MultiHeadAttention, Dense, Dropout, LayerNormalization, Flatten
    7. from tensorflow.python.keras.optimizers import adam_v2
    8.  
    9. # 读取数据
    10. data = pd.read_csv('data.csv')
    11. data['time'] = pd.to_datetime(data['time'], format='%b-%y')
    12.  
    13. n = 6
    14. m = 2
    15.  
    16. # 创建滞后期特征
    17. for i in range(n, 0, -1):
    18.     data[f'lag_{i}'] = data['incidence'].shift(n - i + 1)
    19.  
    20. data = data.dropna().reset_index(drop=True)
    21.  
    22. train_data = data[(data['time'] >= '2004-01-01') & (data['time'] <= '2011-12-31')]
    23. validation_data = data[(data['time'] >= '2012-01-01') & (data['time'] <= '2012-12-31')]
    24.  
    25. # 准备训练数据
    26. X_train = []
    27. y_train = []
    28.  
    29. for i in range(len(train_data) - n - m + 1):
    30.     X_train.append(train_data.iloc[i+n-1][[f'lag_{j}' for j in range(1, n+1)]].values)
    31.     y_train.append(train_data.iloc[i+n:i+n+m]['incidence'].values)
    32.  
    33. X_train = np.array(X_train)
    34. y_train = np.array(y_train)
    35. X_train = X_train.astype(np.float32)
    36. y_train = y_train.astype(np.float32)
    37.  
    38. # 构建Transformer模型
    39. inputs = Input(shape=(n, 1))
    40.  
    41. x = MultiHeadAttention(num_heads=8, key_dim=64)(inputs, inputs)
    42. x = Dropout(0.1)(x)
    43. x = LayerNormalization(epsilon=1e-6)(x + inputs)
    44.  
    45. x = Flatten()(x) # 新增的Flatten层
    46. x = Dense(50, activation='relu')(x)
    47. x = Dropout(0.1)(x)
    48. outputs = Dense(m)(x)
    49.  
    50. model = Model(inputs=inputs, outputs=outputs)
    51.  
    52. model.compile(optimizer=adam_v2.Adam(learning_rate=0.001), loss='mse')
    53.  
    54. # 训练模型
    55. model.fit(X_train, y_train, epochs=200, batch_size=32, verbose=0)
    56.  
    57. def transformer_rolling_forecast(data, model, n, m):
    58.     y_pred = []
    59.  
    60.     for i in range(len(data) - n):
    61.         input_data = data.iloc[i+n-1][[f'lag_{j}' for j in range(1, n+1)]].values.astype(np.float32).reshape(1, n, 1)
    62.         pred = model.predict(input_data)
    63.         y_pred.extend(pred[0])
    64.  
    65.     for i in range(1, m):
    66.         for j in range(len(y_pred) - i):
    67.             y_pred[j+i] = (y_pred[j+i] + y_pred[j]) / 2
    68.  
    69.     return np.array(y_pred)
    70.  
    71. # Predict for train_data and validation_data
    72. y_train_pred_transformer = transformer_rolling_forecast(train_data, model, n, m)[:len(y_train)]
    73. y_validation_pred_transformer = transformer_rolling_forecast(validation_data, model, n, m)[:len(validation_data) - n]
    74.  
    75. # Calculate performance metrics for train_data
    76. mae_train = mean_absolute_error(train_data['incidence'].values[n:len(y_train_pred_transformer)+n], y_train_pred_transformer)
    77. mape_train = np.mean(np.abs((train_data['incidence'].values[n:len(y_train_pred_transformer)+n] - y_train_pred_transformer) / train_data['incidence'].values[n:len(y_train_pred_transformer)+n]))
    78. mse_train = mean_squared_error(train_data['incidence'].values[n:len(y_train_pred_transformer)+n], y_train_pred_transformer)
    79. rmse_train = np.sqrt(mse_train)
    80.  
    81. # Calculate performance metrics for validation_data
    82. mae_validation = mean_absolute_error(validation_data['incidence'].values[n:len(y_validation_pred_transformer)+n], y_validation_pred_transformer)
    83. mape_validation = np.mean(np.abs((validation_data['incidence'].values[n:len(y_validation_pred_transformer)+n] - y_validation_pred_transformer) / validation_data['incidence'].values[n:len(y_validation_pred_transformer)+n]))
    84. mse_validation = mean_squared_error(validation_data['incidence'].values[n:len(y_validation_pred_transformer)+n], y_validation_pred_transformer)
    85. rmse_validation = np.sqrt(mse_validation)
    86.  
    87. print("训练集：", mae_train, mape_train, mse_train, rmse_train)
    88. print("验证集：", mae_validation, mape_validation, mse_validation, rmse_validation)

    结果：

    （4）多步滚动预测-vol. 2

    1. import pandas as pd
    2. import numpy as np
    3. from sklearn.model_selection import train_test_split
    4. from sklearn.metrics import mean_absolute_error, mean_squared_error
    5. from tensorflow.python.keras.models import Sequential, Model
    6. from tensorflow.python.keras.layers import Dense, Conv1D, Flatten, MaxPooling1D, Input, MultiHeadAttention, LayerNormalization, Dropout
    7. from tensorflow.python.keras.optimizers import adam_v2
    8.  
    9. # Loading and preprocessing the data
    10. data = pd.read_csv('data.csv')
    11. data['time'] = pd.to_datetime(data['time'], format='%b-%y')
    12.  
    13. n = 6
    14. m = 2
    15.  
    16. # 创建滞后期特征
    17. for i in range(n, 0, -1):
    18.     data[f'lag_{i}'] = data['incidence'].shift(n - i + 1)
    19.  
    20. data = data.dropna().reset_index(drop=True)
    21.  
    22. train_data = data[(data['time'] >= '2004-01-01') & (data['time'] <= '2011-12-31')]
    23. validation_data = data[(data['time'] >= '2012-01-01') & (data['time'] <= '2012-12-31')]
    24.  
    25. # 只对X_train、y_train、X_validation取奇数行
    26. X_train = train_data[[f'lag_{i}' for i in range(1, n+1)]].iloc[::2].reset_index(drop=True).values
    27. X_train = X_train.reshape(X_train.shape[0], X_train.shape[1], 1)
    28.  
    29. y_train_list = [train_data['incidence'].shift(-i) for i in range(m)]
    30. y_train = pd.concat(y_train_list, axis=1)
    31. y_train.columns = [f'target_{i+1}' for i in range(m)]
    32. y_train = y_train.iloc[::2].reset_index(drop=True).dropna().values[:, 0]
    33.  
    34. X_validation = validation_data[[f'lag_{i}' for i in range(1, n+1)]].iloc[::2].reset_index(drop=True).values
    35. X_validation = X_validation.reshape(X_validation.shape[0], X_validation.shape[1], 1)
    36.  
    37. y_validation = validation_data['incidence'].values
    38.  
    39. # Building the Transformer model
    40. inputs = Input(shape=(n, 1))
    41. x = MultiHeadAttention(num_heads=8, key_dim=64)(inputs, inputs)
    42. x = Dropout(0.1)(x)
    43. x = LayerNormalization(epsilon=1e-6)(x + inputs)
    44. x = Flatten()(x)
    45. x = Dense(50, activation='relu')(x)
    46. outputs = Dense(1)(x)
    47.  
    48. model = Model(inputs=inputs, outputs=outputs)
    49. optimizer = adam_v2.Adam(learning_rate=0.001)
    50. model.compile(optimizer=optimizer, loss='mse')
    51.  
    52. # Train the model
    53. model.fit(X_train, y_train, epochs=200, batch_size=32, verbose=0)
    54.  
    55. # Predict on validation set
    56. y_validation_pred = model.predict(X_validation).flatten()
    57.  
    58. # Compute metrics for validation set
    59. mae_validation = mean_absolute_error(y_validation[:len(y_validation_pred)], y_validation_pred)
    60. mape_validation = np.mean(np.abs((y_validation[:len(y_validation_pred)] - y_validation_pred) / y_validation[:len(y_validation_pred)]))
    61. mse_validation = mean_squared_error(y_validation[:len(y_validation_pred)], y_validation_pred)
    62. rmse_validation = np.sqrt(mse_validation)
    63.  
    64. # Predict on training set
    65. y_train_pred = model.predict(X_train).flatten()
    66.  
    67. # Compute metrics for training set
    68. mae_train = mean_absolute_error(y_train, y_train_pred)
    69. mape_train = np.mean(np.abs((y_train - y_train_pred) / y_train))
    70. mse_train = mean_squared_error(y_train, y_train_pred)
    71. rmse_train = np.sqrt(mse_train)
    72.  
    73. print("验证集：", mae_validation, mape_validation, mse_validation, rmse_validation)
    74. print("训练集：", mae_train, mape_train, mse_train, rmse_train)

    结果：

    （5）多步滚动预测-vol. 3

    1. import pandas as pd
    2. import numpy as np
    3. from sklearn.metrics import mean_absolute_error, mean_squared_error
    4. from tensorflow.python.keras.models import Sequential, Model
    5. from tensorflow.python.keras.layers import Dense, Flatten, Input, MultiHeadAttention, LayerNormalization, Dropout
    6. from tensorflow.python.keras.optimizers import adam_v2
    7.  
    8. # 数据读取和预处理
    9. data = pd.read_csv('data.csv')
    10. data_y = pd.read_csv('data.csv')
    11. data['time'] = pd.to_datetime(data['time'], format='%b-%y')
    12. data_y['time'] = pd.to_datetime(data_y['time'], format='%b-%y')
    13.  
    14. n = 6
    15.  
    16. for i in range(n, 0, -1):
    17.     data[f'lag_{i}'] = data['incidence'].shift(n - i + 1)
    18.  
    19. data = data.dropna().reset_index(drop=True)
    20. train_data = data[(data['time'] >= '2004-01-01') & (data['time'] <= '2011-12-31')]
    21. X_train = train_data[[f'lag_{i}' for i in range(1, n+1)]]
    22. m = 3
    23.  
    24. X_train_list = []
    25. y_train_list = []
    26.  
    27. for i in range(m):
    28.     X_temp = X_train
    29.     y_temp = data_y['incidence'].iloc[n + i:len(data_y) - m + 1 + i]
    30.     
    31.     X_train_list.append(X_temp)
    32.     y_train_list.append(y_temp)
    33.  
    34. for i in range(m):
    35.     X_train_list[i] = X_train_list[i].iloc[:-(m-1)].values
    36.     X_train_list[i] = X_train_list[i].reshape(X_train_list[i].shape[0], X_train_list[i].shape[1], 1)
    37.     y_train_list[i] = y_train_list[i].iloc[:len(X_train_list[i])].values
    38.  
    39. # 模型训练
    40. models = []
    41. for i in range(m):
    42.     # Building the Transformer model
    43.     inputs = Input(shape=(n, 1))
    44.     x = MultiHeadAttention(num_heads=8, key_dim=64)(inputs, inputs)
    45.     x = Dropout(0.1)(x)
    46.     x = LayerNormalization(epsilon=1e-6)(x + inputs)
    47.     x = Flatten()(x)
    48.     x = Dense(50, activation='relu')(x)
    49.     outputs = Dense(1)(x)
    50.  
    51.     model = Model(inputs=inputs, outputs=outputs)
    52.     optimizer = adam_v2.Adam(learning_rate=0.001)
    53.     model.compile(optimizer=optimizer, loss='mse')
    54.     model.fit(X_train_list[i], y_train_list[i], epochs=200, batch_size=32, verbose=0)
    55.     models.append(model)
    56.  
    57. validation_start_time = train_data['time'].iloc[-1] + pd.DateOffset(months=1)
    58. validation_data = data[data['time'] >= validation_start_time]
    59. X_validation = validation_data[[f'lag_{i}' for i in range(1, n+1)]].values
    60. X_validation = X_validation.reshape(X_validation.shape[0], X_validation.shape[1], 1)
    61.  
    62. y_validation_pred_list = [model.predict(X_validation) for model in models]
    63. y_train_pred_list = [model.predict(X_train_list[i]) for i, model in enumerate(models)]
    64.  
    65. def concatenate_predictions(pred_list):
    66.     concatenated = []
    67.     for j in range(len(pred_list[0])):
    68.         for i in range(m):
    69.             concatenated.append(pred_list[i][j])
    70.     return concatenated
    71.  
    72. y_validation_pred = np.array(concatenate_predictions(y_validation_pred_list))[:len(validation_data['incidence'])]
    73. y_train_pred = np.array(concatenate_predictions(y_train_pred_list))[:len(train_data['incidence']) - m + 1]
    74. y_validation_pred = y_validation_pred.flatten()
    75. y_train_pred = y_train_pred.flatten()
    76.  
    77. mae_validation = mean_absolute_error(validation_data['incidence'], y_validation_pred)
    78. mape_validation = np.mean(np.abs((validation_data['incidence'] - y_validation_pred) / validation_data['incidence']))
    79. mse_validation = mean_squared_error(validation_data['incidence'], y_validation_pred)
    80. rmse_validation = np.sqrt(mse_validation)
    81.  
    82. mae_train = mean_absolute_error(train_data['incidence'][:-(m-1)], y_train_pred)
    83. mape_train = np.mean(np.abs((train_data['incidence'][:-(m-1)] - y_train_pred) / train_data['incidence'][:-(m-1)]))
    84. mse_train = mean_squared_error(train_data['incidence'][:-(m-1)], y_train_pred)
    85. rmse_train = np.sqrt(mse_train)
    86.  
    87. print("验证集：", mae_validation, mape_validation, mse_validation, rmse_validation)
    88. print("训练集：", mae_train, mape_train, mse_train, rmse_train)

    结果：

    三、数据

    链接：https://pan.baidu.com/s/1EFaWfHoG14h15KCEhn1STg?pwd=q41n

    提取码：q41n

  • 相关阅读:
    【C++】构造函数意义 ( 构造函数显式调用与隐式调用 | 构造函数替代方案 - 初始化函数 | 初始化函数缺陷 | 默认构造函数 )
    Github 2024-02-20开源项目日报 Top10
    基于Java毕业设计影院网上售票系统演示录像源码+系统+mysql+lw文档+部署软件
    耳机类型分类
    【定时开关机】windows 10 如何设置定时开关机
    请求传参.
    组件库自定义主题换肤实现方案
    27、Flink 的SQL之SELECT (SQL Hints 和 Joins)介绍及详细示例（2-1）
    GD32F103x 定时器
    实例方法(instance method)、类方法、构造方法（三）
  • 原文地址：https://blog.csdn.net/qq_30452897/article/details/133636381