AutoDL平台transformers环境搭建

AutoDL平台transformers环境搭建

租借GPU可以参考

AutoDL平台租借GPU详解

一、激活base环境

1.进入终端

vim  ~/.bashrc
1

2、然后按英文模式的 i 进入编辑，按键盘下键到最后输入

source root/miniconda3/etc/profile.d/conda.sh
1

3、然后先按键盘Esc键，使用命令 :wq 保存退出

4、输入以下命令刷新

bash
1

5、进入conda环境

conda activate base
# 可能会报错‘...conda init’
#1、执行：
conda init
#2、执行：
bash
#3、重新激活：
conda activate base
1
2
3
4
5
6
7
8

二、创建自己的环境

# 1、创建transformers环境
(base) root@autodl-container-adbc11ae52-f2ebff02:~# conda create -n transformers python=3.9 -y



# 2、此时，会有两个环境
(base) root@autodl-container-adbc11ae52-f2ebff02:~# conda info --envs
# conda environments:
#
base                  *  /root/miniconda3
transformers             /root/miniconda3/envs/transformers

# 3、激活创建的环境
(base) root@autodl-container-adbc11ae52-f2ebff02:~# conda activate transformers
(transformers) root@autodl-container-adbc11ae52-f2ebff02:~# 


# 4、安装pytorch

# 4.1 查看机器支持的cuda版本，可以看到最高支持到CUDA Version: 12.0
(transformers) root@autodl-container-adbc11ae52-f2ebff02:~# nvidia-smi 
Sat Oct 14 11:40:39 2023       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.89.02    Driver Version: 525.89.02    CUDA Version: 12.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  On   | 00000000:3D:00.0 Off |                  N/A |
| 30%   32C    P8    19W / 250W |     14MiB / 11264MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
+-----------------------------------------------------------------------------+

# 4.2 设置下载源为清华源
pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple


# 4.3 进入pytorch官网下载最新版pytorch
# https://pytorch.org/
# 推荐使用pip安装

pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50

# 4.4 测试是否安装成功
(transformers) root@autodl-container-adbc11ae52-f2ebff02:~# python
Python 3.9.18 (main, Sep 11 2023, 13:41:44) 
[GCC 11.2.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> 
>>> print(torch.__version__)
2.1.0+cu118
>>> print(torch.cuda.is_available())
True
>>> 


# 5、安装transformers相关库
pip install transformers datasets evaluate peft accelerate gradio optimum sentencepiece
pip install scikit-learn pandas matplotlib tensorboard nltk rouge

# 6、验证安装是否成功
>>> from transformers import *
>>> 



# 7、虚拟环境添加到可选的kernel
conda install ipykernel
ipython kernel install --user --name=transformers
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

此时我们在jupyter上刷新页面，就能看到自己刚装的环境

以后使用的时候，可以选择此环境