-
Meta-Llama-3-8B-Instruct本地推理
Meta-Llama-3-8B-Instruct 本地推理
基础环境信息（wsl2安装Ubuntu22.04 + miniconda）
使用miniconda搭建环境
(base) :~$ conda create --name pytorch212 python=3.10 Retrieving notices: ...working... done Channels: - defaults Platform: linux-64 Collecting package metadata (repodata.json): done Solving environment: done ## Package Plan ## environment location: /home/chop/miniconda3/envs/pytorch212 added / updated specs: - python=3.10 The following NEW packages will be INSTALLED: _libgcc_mutex anaconda/pkgs/main/linux-64::_libgcc_mutex-0.1-main _openmp_mutex anaconda/pkgs/main/linux-64::_openmp_mutex-5.1-1_gnu bzip2 anaconda/pkgs/main/linux-64::bzip2-1.0.8-h5eee18b_5 ca-certificates anaconda/pkgs/main/linux-64::ca-certificates-2024.3.11-h06a4308_0 ld_impl_linux-64 anaconda/pkgs/main/linux-64::ld_impl_linux-64-2.38-h1181459_1 libffi anaconda/pkgs/main/linux-64::libffi-3.4.4-h6a678d5_0 libgcc-ng anaconda/pkgs/main/linux-64::libgcc-ng-11.2.0-h1234567_1 libgomp anaconda/pkgs/main/linux-64::libgomp-11.2.0-h1234567_1 libstdcxx-ng anaconda/pkgs/main/linux-64::libstdcxx-ng-11.2.0-h1234567_1 libuuid anaconda/pkgs/main/linux-64::libuuid-1.41.5-h5eee18b_0 ncurses anaconda/pkgs/main/linux-64::ncurses-6.4-h6a678d5_0 openssl anaconda/pkgs/main/linux-64::openssl-3.0.13-h7f8727e_0 pip anaconda/pkgs/main/linux-64::pip-23.3.1-py310h06a4308_0 python anaconda/pkgs/main/linux-64::python-3.10.14-h955ad1f_0 readline anaconda/pkgs/main/linux-64::readline-8.2-h5eee18b_0 setuptools anaconda/pkgs/main/linux-64::setuptools-68.2.2-py310h06a4308_0 sqlite anaconda/pkgs/main/linux-64::sqlite-3.41.2-h5eee18b_0 tk anaconda/pkgs/main/linux-64::tk-8.6.12-h1ccaba5_0 tzdata anaconda/pkgs/main/noarch::tzdata-2024a-h04d1e81_0 wheel anaconda/pkgs/main/linux-64::wheel-0.41.2-py310h06a4308_0 xz anaconda/pkgs/main/linux-64::xz-5.4.6-h5eee18b_0 zlib anaconda/pkgs/main/linux-64::zlib-1.2.13-h5eee18b_0 Proceed ([y]/n)? y Downloading and Extracting Packages: Preparing transaction: done Verifying transaction: done Executing transaction: done # # To activate this environment, use # # $ conda activate pytorch212 # # To deactivate an active environment, use # # $ conda deactivate (base) :~$ conda activate pytorch212- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
- 42
- 43
- 44
- 45
- 46
- 47
- 48
- 49
- 50
- 51
- 52
- 53
- 54
- 55
- 56
- 57
- 58
- 59
- 60
- 61
- 62
下载Llama-Chinese
(pytorch212) :~$git init (pytorch212) :~$git clone https://github.com/LlamaFamily/Llama-Chinese.git (pytorch212) :~$ cd Llama-Chinese (pytorch212) :~/Llama-Chinese$ ls README.md README_EN.md assets data docker docs examples inference-speed requirements.txt scripts train- 1
- 2
- 3
- 4
- 5
安装所需要的软件包
(pytorch212) :~/Llama-Chinese/Llama-Chinese$ pip install -r requirements.txt Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple Collecting torch==2.1.2 (from -r requirements.txt (line 1)) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/03/f1/13137340776dd5d5bcfd2574c9c6dfcc7618285035cd77240496e5c1a79b/torch-2.1.2-cp310-cp310-manylinux1_x86_64.whl (670.2 MB) Requirement already satisfied: bitsandbytes==0.42.0 in /home/chop/.local/lib/python3.10/site-packages (from -r requirements.txt (line 2)) (0.42.0) Requirement already satisfied: accelerate==0.27.2 in /home/chop/.local/lib/python3.10/site-packages (from -r requirements.txt (line 3)) (0.27.2) Requirement already satisfied: numpy==1.26.4 in /home/chop/.local/lib/python3.10/site-packages (from -r requirements.txt (line 4)) (1.26.4) Requirement already satisfied: gekko==1.0.6 in /home/chop/.local/lib/python3.10/site-packages (from -r requirements.txt (line 5)) (1.0.6) Requirement already satisfied: pandas in /home/chop/.local/lib/python3.10/site-packages (from -r requirements.txt (line 6)) (2.2.2) Requirement already satisfied: scipy in /home/chop/.local/lib/python3.10/site-packages (from -r requirements.txt (line 7)) (1.13.0) Requirement already satisfied: sentencepiece==0.2.0 in /home/chop/.local/lib/python3.10/site-packages (from -r requirements.txt (line 8)) (0.2.0) Requirement already satisfied: datasets in /home/chop/.local/lib/python3.10/site-packages (from -r requirements.txt (line 9)) (2.19.0) Requirement already satisfied: evaluate in /home/chop/.local/lib/python3.10/site-packages (from -r requirements.txt (line 10)) (0.4.1) Requirement already satisfied: pytest in /home/chop/.local/lib/python3.10/site-packages (from -r requirements.txt (line 11)) (8.1.1) Requirement already satisfied: peft==0.8.2 in /home/chop/.local/lib/python3.10/site-packages (from -r requirements.txt (line 12)) (0.8.2) Requirement already satisfied: transformers==4.39.0 in /home/chop/.local/lib/python3.10/site-packages (from -r requirements.txt (line 13)) (4.39.0) Collecting deepspeed==0.14.0 (from -r requirements.txt (line 14)) Using cached deepspeed-0.14.0-py3-none-any.whl Requirement already satisfied: scikit-learn in /home/chop/.local/lib/python3.10/site-packages (from -r requirements.txt (line 15)) (1.4.2) Requirement already satisfied: torchvision in /home/chop/.local/lib/python3.10/site-packages (from -r requirements.txt (line 16)) (0.17.2) Requirement already satisfied: torchdata in /home/chop/.local/lib/python3.10/site-packages (from -r requirements.txt (line 17)) (0.7.1) Requirement already satisfied: torchaudio in /home/chop/.local/lib/python3.10/site-packages (from -r requirements.txt (line 18)) (2.2.2) Requirement already satisfied: tensorboard in /home/chop/.local/lib/python3.10/site-packages (from -r requirements.txt (line 19)) (2.16.2) Requirement already satisfied: gradio in /home/chop/.local/lib/python3.10/site-packages (from -r requirements.txt (line 20)) (4.27.0) Requirement already satisfied: packaging in /home/chop/.local/lib/python3.10/site-packages (from -r requirements.txt (line 21)) (24.0) Requirement already satisfied: filelock in /home/chop/.local/lib/python3.10/site-packages (from torch==2.1.2->-r requirements.txt (line 1)) (3.13.4) Requirement already satisfied: typing-extensions in /home/chop/.local/lib/python3.10/site-packages (from torch==2.1.2->-r requirements.txt (line 1)) (4.11.0) Requirement already satisfied: sympy in /home/chop/.local/lib/python3.10/site-packages (from torch==2.1.2->-r requirements.txt (line 1)) (1.12) Requirement already satisfied: networkx in /home/chop/.local/lib/python3.10/site-packages (from torch==2.1.2->-r requirements.txt (line 1)) (3.3) Requirement already satisfied: jinja2 in /home/chop/.local/lib/python3.10/site-packages (from torch==2.1.2->-r requirements.txt (line 1)) (3.1.3) Requirement already satisfied: fsspec in /home/chop/.local/lib/python3.10/site-packages (from torch==2.1.2->-r requirements.txt (line 1)) (2024.3.1) Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.1.105 in /home/chop/.local/lib/python3.10/site-packages (from torch==2.1.2->-r requirements.txt (line 1)) (12.1.105) Requirement already satisfied: nvidia-cuda-runtime-cu12==12.1.105 in /home/chop/.local/lib/python3.10/site-packages (from torch==2.1.2->-r requirements.txt (line 1)) (12.1.105) Requirement already satisfied: nvidia-cuda-cupti-cu12==12.1.105 in /home/chop/.local/lib/python3.10/site-packages (from torch==2.1.2->-r requirements.txt (line 1)) (12.1.105) Requirement already satisfied: nvidia-cudnn-cu12==8.9.2.26 in /home/chop/.local/lib/python3.10/site-packages (from torch==2.1.2->-r requirements.txt (line 1)) (8.9.2.26) Requirement already satisfied: nvidia-cublas-cu12==12.1.3.1 in /home/chop/.local/lib/python3.10/site-packages (from torch==2.1.2->-r requirements.txt (line 1)) (12.1.3.1) Requirement already satisfied: nvidia-cufft-cu12==11.0.2.54 in /home/chop/.local/lib/python3.10/site-packages (from torch==2.1.2->-r requirements.txt (line 1)) (11.0.2.54) Requirement already satisfied: nvidia-curand-cu12==10.3.2.106 in /home/chop/.local/lib/python3.10/site-packages (from torch==2.1.2->-r requirements.txt (line 1)) (10.3.2.106) Requirement already satisfied: nvidia-cusolver-cu12==11.4.5.107 in /home/chop/.local/lib/python3.10/site-packages (from torch==2.1.2->-r requirements.txt (line 1)) (11.4.5.107) Requirement already satisfied: nvidia-cusparse-cu12==12.1.0.106 in /home/chop/.local/lib/python3.10/site-packages (from torch==2.1.2->-r requirements.txt (line 1)) (12.1.0.106) Collecting nvidia-nccl-cu12==2.18.1 (from torch==2.1.2->-r requirements.txt (line 1)) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/a4/05/23f8f38eec3d28e4915725b233c24d8f1a33cb6540a882f7b54be1befa02/nvidia_nccl_cu12-2.18.1-py3-none-manylinux1_x86_64.whl (209.8 MB) Requirement already satisfied: nvidia-nvtx-cu12==12.1.105 in /home/chop/.local/lib/python3.10/site-packages (from torch==2.1.2->-r requirements.txt (line 1)) (12.1.105) Collecting triton==2.1.0 (from torch==2.1.2->-r requirements.txt (line 1)) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/4d/22/91a8af421c8a8902dde76e6ef3db01b258af16c53d81e8c0d0dc13900a9e/triton-2.1.0-0-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (89.2 MB) Requirement already satisfied: psutil in /home/chop/.local/lib/python3.10/site-packages (from accelerate==0.27.2->-r requirements.txt (line 3)) (5.9.8) Collecting pyyaml (from accelerate==0.27.2->-r requirements.txt (line 3)) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/29/61/bf33c6c85c55bc45a29eee3195848ff2d518d84735eb0e2d8cb42e0d285e/PyYAML-6.0.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (705 kB) Requirement already satisfied: huggingface-hub in /home/chop/.local/lib/python3.10/site-packages (from accelerate==0.27.2->-r requirements.txt (line 3)) (0.22.2) Requirement already satisfied: safetensors>=0.3.1 in /home/chop/.local/lib/python3.10/site-packages (from accelerate==0.27.2->-r requirements.txt (line 3)) (0.4.3) Requirement already satisfied: tqdm in /home/chop/.local/lib/python3.10/site-packages (from peft==0.8.2->-r requirements.txt (line 12)) (4.66.2) Requirement already satisfied: regex!=2019.12.17 in /home/chop/.local/lib/python3.10/site-packages (from transformers==4.39.0->-r requirements.txt (line 13)) (2024.4.16) Requirement already satisfied: requests in /home/chop/.local/lib/python3.10/site-packages (from transformers==4.39.0->-r requirements.txt (line 13)) (2.31.0) Requirement already satisfied: tokenizers<0.19,>=0.14 in /home/chop/.local/lib/python3.10/site-packages (from transformers==4.39.0->-r requirements.txt (line 13)) (0.15.2) Collecting hjson (from deepspeed==0.14.0->-r requirements.txt (line 14)) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/1f/7f/13cd798d180af4bf4c0ceddeefba2b864a63c71645abc0308b768d67bb81/hjson-3.1.0-py3-none-any.whl (54 kB) Collecting ninja (from deepspeed==0.14.0->-r requirements.txt (line 14)) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/6d/92/8d7aebd4430ab5ff65df2bfee6d5745f95c004284db2d8ca76dcbfd9de47/ninja-1.11.1.1-py2.py3-none-manylinux1_x86_64.manylinux_2_5_x86_64.whl (307 kB) Collecting py-cpuinfo (from deepspeed==0.14.0->-r requirements.txt (line 14)) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/e0/a9/023730ba63db1e494a271cb018dcd361bd2c917ba7004c3e49d5daf795a2/py_cpuinfo-9.0.0-py3-none-any.whl (22 kB) Requirement already satisfied: pydantic in /home/chop/.local/lib/python3.10/site-packages (from deepspeed==0.14.0->-r requirements.txt (line 14)) (2.7.0) Collecting pynvml (from deepspeed==0.14.0->-r requirements.txt (line 14)) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/5b/9c/adb8070059caaa15d5a572b66bccd95900d8c1b9fa54d6ecea6ae97448d1/pynvml-11.5.0-py3-none-any.whl (53 kB) Requirement already satisfied: nvidia-nvjitlink-cu12 in /home/chop/.local/lib/python3.10/site-packages (from nvidia-cusolver-cu12==11.4.5.107->torch==2.1.2->-r requirements.txt (line 1)) (12.4.127) Requirement already satisfied: python-dateutil>=2.8.2 in /home/chop/.local/lib/python3.10/site-packages (from pandas->-r requirements.txt (line 6)) (2.9.0.post0) Requirement already satisfied: pytz>=2020.1 in /home/chop/.local/lib/python3.10/site-packages (from pandas->-r requirements.txt (line 6)) (2024.1) Requirement already satisfied: tzdata>=2022.7 in /home/chop/.local/lib/python3.10/site-packages (from pandas->-r requirements.txt (line 6)) (2024.1) Requirement already satisfied: pyarrow>=12.0.0 in /home/chop/.local/lib/python3.10/site-packages (from datasets->-r requirements.txt (line 9)) (16.0.0) Requirement already satisfied: pyarrow-hotfix in /home/chop/.local/lib/python3.10/site-packages (from datasets->-r requirements.txt (line 9)) (0.6) Requirement already satisfied: dill<0.3.9,>=0.3.0 in /home/chop/.local/lib/python3.10/site-packages (from datasets->-r requirements.txt (line 9)) (0.3.8) Requirement already satisfied: xxhash in /home/chop/.local/lib/python3.10/site-packages (from datasets->-r requirements.txt (line 9)) (3.4.1) Requirement already satisfied: multiprocess in /home/chop/.local/lib/python3.10/site-packages (from datasets->-r requirements.txt (line 9)) (0.70.16) Requirement already satisfied: aiohttp in /home/chop/.local/lib/python3.10/site-packages (from datasets->-r requirements.txt (line 9)) (3.9.5) Requirement already satisfied: responses<0.19 in /home/chop/.local/lib/python3.10/site-packages (from evaluate->-r requirements.txt (line 10)) (0.18.0) Requirement already satisfied: iniconfig in /home/chop/.local/lib/python3.10/site-packages (from pytest->-r requirements.txt (line 11)) (2.0.0) Requirement already satisfied: pluggy<2.0,>=1.4 in /home/chop/.local/lib/python3.10/site-packages (from pytest->-r requirements.txt (line 11)) (1.5.0) Requirement already satisfied: exceptiongroup>=1.0.0rc8 in /home/chop/.local/lib/python3.10/site-packages (from pytest->-r requirements.txt (line 11)) (1.2.1) Requirement already satisfied: tomli>=1 in /home/chop/.local/lib/python3.10/site-packages (from pytest->-r requirements.txt (line 11)) (2.0.1) Requirement already satisfied: joblib>=1.2.0 in /home/chop/.local/lib/python3.10/site-packages (from scikit-learn->-r requirements.txt (line 15)) (1.4.0) Requirement already satisfied: threadpoolctl>=2.0.0 in /home/chop/.local/lib/python3.10/site-packages (from scikit-learn->-r requirements.txt (line 15)) (3.4.0) INFO: pip is looking at multiple versions of torchvision to determine which version is compatible with other requirements. This could take a while. Collecting torchvision (from -r requirements.txt (line 16)) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/1a/e8/13432ae6be567b577a4c89d1bd50084e4d989b379a7be8050380b5ab3a6e/torchvision-0.17.1-cp310-cp310-manylinux1_x86_64.whl (6.9 MB) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/d8/51/55393d57c2d95311b1675c8cd37d307f5022460cf98746e4df882dfb415c/torchvision-0.17.0-cp310-cp310-manylinux1_x86_64.whl (6.9 MB) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/da/ae/76bd3682465730dea7be21f36a8160a911a470de6f26228904f222e7fefe/torchvision-0.16.2-cp310-cp310-manylinux1_x86_64.whl (6.8 MB) Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /home/chop/.local/lib/python3.10/site-packages (from torchvision->-r requirements.txt (line 16)) (10.3.0) Requirement already satisfied: urllib3>=1.25 in /home/chop/.local/lib/python3.10/site-packages (from torchdata->-r requirements.txt (line 17)) (2.2.1) INFO: pip is looking at multiple versions of torchaudio to determine which version is compatible with other requirements. This could take a while. Collecting torchaudio (from -r requirements.txt (line 18)) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/ce/99/7485966a902905e206eda57fdca8de69545c107e33eefe9f6536c2dda16c/torchaudio-2.2.1-cp310-cp310-manylinux1_x86_64.whl (3.3 MB) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/30/fc/cdcf7c2071539ea147ddb6de2b538d9c1599665b621f2e6cf0b3ef51d20d/torchaudio-2.2.0-cp310-cp310-manylinux1_x86_64.whl (3.3 MB) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/25/83/4170df23c16c25818ce9591fd2b8109cda3f725d115d77417d4efef4eb46/torchaudio-2.1.2-cp310-cp310-manylinux1_x86_64.whl (3.3 MB) Requirement already satisfied: absl-py>=0.4 in /home/chop/.local/lib/python3.10/site-packages (from tensorboard->-r requirements.txt (line 19)) (2.1.0) Requirement already satisfied: grpcio>=1.48.2 in /home/chop/.local/lib/python3.10/site-packages (from tensorboard->-r requirements.txt (line 19)) (1.62.2) Requirement already satisfied: markdown>=2.6.8 in /home/chop/.local/lib/python3.10/site-packages (from tensorboard->-r requirements.txt (line 19)) (3.6) Requirement already satisfied: protobuf!=4.24.0,>=3.19.6 in /home/chop/.local/lib/python3.10/site-packages (from tensorboard->-r requirements.txt (line 19)) (5.26.1) Requirement already satisfied: setuptools>=41.0.0 in /home/chop/miniconda3/envs/pytorch212/lib/python3.10/site-packages (from tensorboard->-r requirements.txt (line 19)) (68.2.2) Collecting six>1.9 (from tensorboard->-r requirements.txt (line 19)) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/d9/5a/e7c31adbe875f2abbb91bd84cf2dc52d792b5a01506781dbcf25c91daf11/six-1.16.0-py2.py3-none-any.whl (11 kB) Requirement already satisfied: tensorboard-data-server<0.8.0,>=0.7.0 in /home/chop/.local/lib/python3.10/site-packages (from tensorboard->-r requirements.txt (line 19)) (0.7.2) Requirement already satisfied: werkzeug>=1.0.1 in /home/chop/.local/lib/python3.10/site-packages (from tensorboard->-r requirements.txt (line 19)) (3.0.2) Requirement already satisfied: aiofiles<24.0,>=22.0 in /home/chop/.local/lib/python3.10/site-packages (from gradio->-r requirements.txt (line 20)) (23.2.1) Requirement already satisfied: altair<6.0,>=4.2.0 in /home/chop/.local/lib/python3.10/site-packages (from gradio->-r requirements.txt (line 20)) (5.3.0) Requirement already satisfied: fastapi in /home/chop/.local/lib/python3.10/site-packages (from gradio->-r requirements.txt (line 20)) (0.110.2) Requirement already satisfied: ffmpy in /home/chop/.local/lib/python3.10/site-packages (from gradio->-r requirements.txt (line 20)) (0.3.2) Requirement already satisfied: gradio-client==0.15.1 in /home/chop/.local/lib/python3.10/site-packages (from gradio->-r requirements.txt (line 20)) (0.15.1) Requirement already satisfied: httpx>=0.24.1 in /home/chop/.local/lib/python3.10/site-packages (from gradio->-r requirements.txt (line 20)) (0.27.0) Requirement already satisfied: importlib-resources<7.0,>=1.3 in /home/chop/.local/lib/python3.10/site-packages (from gradio->-r requirements.txt (line 20)) (6.4.0) Requirement already satisfied: markupsafe~=2.0 in /home/chop/.local/lib/python3.10/site-packages (from gradio->-r requirements.txt (line 20)) (2.1.5) Requirement already satisfied: matplotlib~=3.0 in /home/chop/.local/lib/python3.10/site-packages (from gradio->-r requirements.txt (line 20)) (3.8.4) Requirement already satisfied: orjson~=3.0 in /home/chop/.local/lib/python3.10/site-packages (from gradio->-r requirements.txt (line 20)) (3.10.1) Requirement already satisfied: pydub in /home/chop/.local/lib/python3.10/site-packages (from gradio->-r requirements.txt (line 20)) (0.25.1) Requirement already satisfied: python-multipart>=0.0.9 in /home/chop/.local/lib/python3.10/site-packages (from gradio->-r requirements.txt (line 20)) (0.0.9) Requirement already satisfied: ruff>=0.2.2 in /home/chop/.local/lib/python3.10/site-packages (from gradio->-r requirements.txt (line 20)) (0.4.1) Requirement already satisfied: semantic-version~=2.0 in /home/chop/.local/lib/python3.10/site-packages (from gradio->-r requirements.txt (line 20)) (2.10.0) Requirement already satisfied: tomlkit==0.12.0 in /home/chop/.local/lib/python3.10/site-packages (from gradio->-r requirements.txt (line 20)) (0.12.0) Requirement already satisfied: typer<1.0,>=0.12 in /home/chop/.local/lib/python3.10/site-packages (from gradio->-r requirements.txt (line 20)) (0.12.3) Requirement already satisfied: uvicorn>=0.14.0 in /home/chop/.local/lib/python3.10/site-packages (from gradio->-r requirements.txt (line 20)) (0.29.0) Requirement already satisfied: websockets<12.0,>=10.0 in /home/chop/.local/lib/python3.10/site-packages (from gradio-client==0.15.1->gradio->-r requirements.txt (line 20)) (11.0.3) Requirement already satisfied: jsonschema>=3.0 in /home/chop/.local/lib/python3.10/site-packages (from altair<6.0,>=4.2.0->gradio->-r requirements.txt (line 20)) (4.21.1) Requirement already satisfied: toolz in /home/chop/.local/lib/python3.10/site-packages (from altair<6.0,>=4.2.0->gradio->-r requirements.txt (line 20)) (0.12.1) Requirement already satisfied: aiosignal>=1.1.2 in /home/chop/.local/lib/python3.10/site-packages (from aiohttp->datasets->-r requirements.txt (line 9)) (1.3.1) Requirement already satisfied: attrs>=17.3.0 in /home/chop/.local/lib/python3.10/site-packages (from aiohttp->datasets->-r requirements.txt (line 9)) (23.2.0) Requirement already satisfied: frozenlist>=1.1.1 in /home/chop/.local/lib/python3.10/site-packages (from aiohttp->datasets->-r requirements.txt (line 9)) (1.4.1) Requirement already satisfied: multidict<7.0,>=4.5 in /home/chop/.local/lib/python3.10/site-packages (from aiohttp->datasets->-r requirements.txt (line 9)) (6.0.5) Requirement already satisfied: yarl<2.0,>=1.0 in /home/chop/.local/lib/python3.10/site-packages (from aiohttp->datasets->-r requirements.txt (line 9)) (1.9.4) Requirement already satisfied: async-timeout<5.0,>=4.0 in /home/chop/.local/lib/python3.10/site-packages (from aiohttp->datasets->-r requirements.txt (line 9)) (4.0.3) Requirement already satisfied: anyio in /home/chop/.local/lib/python3.10/site-packages (from httpx>=0.24.1->gradio->-r requirements.txt (line 20)) (4.3.0) Requirement already satisfied: certifi in /home/chop/.local/lib/python3.10/site-packages (from httpx>=0.24.1->gradio->-r requirements.txt (line 20)) (2024.2.2) Requirement already satisfied: httpcore==1.* in /home/chop/.local/lib/python3.10/site-packages (from httpx>=0.24.1->gradio->-r requirements.txt (line 20)) (1.0.5) Requirement already satisfied: idna in /home/chop/.local/lib/python3.10/site-packages (from httpx>=0.24.1->gradio->-r requirements.txt (line 20)) (3.7) Requirement already satisfied: sniffio in /home/chop/.local/lib/python3.10/site-packages (from httpx>=0.24.1->gradio->-r requirements.txt (line 20)) (1.3.1) Requirement already satisfied: h11<0.15,>=0.13 in /home/chop/.local/lib/python3.10/site-packages (from httpcore==1.*->httpx>=0.24.1->gradio->-r requirements.txt (line 20)) (0.14.0) Requirement already satisfied: contourpy>=1.0.1 in /home/chop/.local/lib/python3.10/site-packages (from matplotlib~=3.0->gradio->-r requirements.txt (line 20)) (1.2.1) Requirement already satisfied: cycler>=0.10 in /home/chop/.local/lib/python3.10/site-packages (from matplotlib~=3.0->gradio->-r requirements.txt (line 20)) (0.12.1) Requirement already satisfied: fonttools>=4.22.0 in /home/chop/.local/lib/python3.10/site-packages (from matplotlib~=3.0->gradio->-r requirements.txt (line 20)) (4.51.0) Requirement already satisfied: kiwisolver>=1.3.1 in /home/chop/.local/lib/python3.10/site-packages (from matplotlib~=3.0->gradio->-r requirements.txt (line 20)) (1.4.5) Collecting pyparsing>=2.3.1 (from matplotlib~=3.0->gradio->-r requirements.txt (line 20)) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/9d/ea/6d76df31432a0e6fdf81681a895f009a4bb47b3c39036db3e1b528191d52/pyparsing-3.1.2-py3-none-any.whl (103 kB) Requirement already satisfied: annotated-types>=0.4.0 in /home/chop/.local/lib/python3.10/site-packages (from pydantic->deepspeed==0.14.0->-r requirements.txt (line 14)) (0.6.0) Requirement already satisfied: pydantic-core==2.18.1 in /home/chop/.local/lib/python3.10/site-packages (from pydantic->deepspeed==0.14.0->-r requirements.txt (line 14)) (2.18.1) Requirement already satisfied: charset-normalizer<4,>=2 in /home/chop/.local/lib/python3.10/site-packages (from requests->transformers==4.39.0->-r requirements.txt (line 13)) (3.3.2) Requirement already satisfied: click>=8.0.0 in /home/chop/.local/lib/python3.10/site-packages (from typer<1.0,>=0.12->gradio->-r requirements.txt (line 20)) (8.1.7) Requirement already satisfied: shellingham>=1.3.0 in /home/chop/.local/lib/python3.10/site-packages (from typer<1.0,>=0.12->gradio->-r requirements.txt (line 20)) (1.5.4) Requirement already satisfied: rich>=10.11.0 in /home/chop/.local/lib/python3.10/site-packages (from typer<1.0,>=0.12->gradio->-r requirements.txt (line 20)) (13.7.1) Requirement already satisfied: starlette<0.38.0,>=0.37.2 in /home/chop/.local/lib/python3.10/site-packages (from fastapi->gradio->-r requirements.txt (line 20)) (0.37.2) Requirement already satisfied: mpmath>=0.19 in /home/chop/.local/lib/python3.10/site-packages (from sympy->torch==2.1.2->-r requirements.txt (line 1)) (1.3.0) Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /home/chop/.local/lib/python3.10/site-packages (from jsonschema>=3.0->altair<6.0,>=4.2.0->gradio->-r requirements.txt (line 20)) (2023.12.1) Requirement already satisfied: referencing>=0.28.4 in /home/chop/.local/lib/python3.10/site-packages (from jsonschema>=3.0->altair<6.0,>=4.2.0->gradio->-r requirements.txt (line 20)) (0.34.0) Requirement already satisfied: rpds-py>=0.7.1 in /home/chop/.local/lib/python3.10/site-packages (from jsonschema>=3.0->altair<6.0,>=4.2.0->gradio->-r requirements.txt (line 20)) (0.18.0) Requirement already satisfied: markdown-it-py>=2.2.0 in /home/chop/.local/lib/python3.10/site-packages (from rich>=10.11.0->typer<1.0,>=0.12->gradio->-r requirements.txt (line 20)) (3.0.0) Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /home/chop/.local/lib/python3.10/site-packages (from rich>=10.11.0->typer<1.0,>=0.12->gradio->-r requirements.txt (line 20)) (2.17.2) Requirement already satisfied: mdurl~=0.1 in /home/chop/.local/lib/python3.10/site-packages (from markdown-it-py>=2.2.0->rich>=10.11.0->typer<1.0,>=0.12->gradio->-r requirements.txt (line 20)) (0.1.2) Installing collected packages: py-cpuinfo, ninja, hjson, triton, six, pyyaml, pyparsing, pynvml, nvidia-nccl-cu12, torch, torchvision, torchaudio, deepspeed Attempting uninstall: triton Found existing installation: triton 2.2.0 Uninstalling triton-2.2.0: Successfully uninstalled triton-2.2.0 Attempting uninstall: nvidia-nccl-cu12 Found existing installation: nvidia-nccl-cu12 2.19.3 Uninstalling nvidia-nccl-cu12-2.19.3: Successfully uninstalled nvidia-nccl-cu12-2.19.3 Attempting uninstall: torch Found existing installation: torch 2.2.2 Uninstalling torch-2.2.2: Successfully uninstalled torch-2.2.2 Attempting uninstall: torchvision Found existing installation: torchvision 0.17.2 Uninstalling torchvision-0.17.2: Successfully uninstalled torchvision-0.17.2 Attempting uninstall: torchaudio Found existing installation: torchaudio 2.2.2 Uninstalling torchaudio-2.2.2: Successfully uninstalled torchaudio-2.2.2 Successfully installed deepspeed-0.14.0 hjson-3.1.0 ninja-1.11.1.1 nvidia-nccl-cu12-2.18.1 py-cpuinfo-9.0.0 pynvml-11.5.0 pyparsing-3.1.2 pyyaml-6.0.1 six-1.16.0 torch-2.1.2 torchaudio-2.1.2 torchvision-0.16.2 triton-2.1.0 (pytorch212) :~/Llama-Chinese/Llama-Chinese$ pip list Package Version ------------------------- ----------- absl-py 2.1.0 accelerate 0.27.2 aiofiles 23.2.1 aiohttp 3.9.5 aiosignal 1.3.1 altair 5.3.0 annotated-types 0.6.0 anyio 4.3.0 async-timeout 4.0.3 attrs 23.2.0 bitsandbytes 0.42.0 certifi 2024.2.2 charset-normalizer 3.3.2 click 8.1.7 contourpy 1.2.1 cycler 0.12.1 datasets 2.19.0 deepspeed 0.14.0 dill 0.3.8 evaluate 0.4.1 exceptiongroup 1.2.1 fastapi 0.110.2 ffmpy 0.3.2 filelock 3.13.4 fonttools 4.51.0 frozenlist 1.4.1 fsspec 2024.3.1 gekko 1.0.6 gradio 4.27.0 gradio_client 0.15.1 grpcio 1.62.2 h11 0.14.0 hjson 3.1.0 httpcore 1.0.5 httpx 0.27.0 huggingface-hub 0.22.2 idna 3.7 importlib_resources 6.4.0 iniconfig 2.0.0 Jinja2 3.1.3 joblib 1.4.0 jsonschema 4.21.1 jsonschema-specifications 2023.12.1 kiwisolver 1.4.5 Markdown 3.6 markdown-it-py 3.0.0 MarkupSafe 2.1.5 matplotlib 3.8.4 mdurl 0.1.2 mpmath 1.3.0 multidict 6.0.5 multiprocess 0.70.16 networkx 3.3 ninja 1.11.1.1 numpy 1.26.4 nvidia-cublas-cu12 12.1.3.1 nvidia-cuda-cupti-cu12 12.1.105 nvidia-cuda-nvrtc-cu12 12.1.105 nvidia-cuda-runtime-cu12 12.1.105 nvidia-cudnn-cu12 8.9.2.26 nvidia-cufft-cu12 11.0.2.54 nvidia-curand-cu12 10.3.2.106 nvidia-cusolver-cu12 11.4.5.107 nvidia-cusparse-cu12 12.1.0.106 nvidia-nccl-cu12 2.18.1 nvidia-nvjitlink-cu12 12.4.127 nvidia-nvtx-cu12 12.1.105 orjson 3.10.1 packaging 24.0 pandas 2.2.2 peft 0.8.2 pillow 10.3.0 pip 23.3.1 pluggy 1.5.0 protobuf 5.26.1 psutil 5.9.8 py-cpuinfo 9.0.0 pyarrow 16.0.0 pyarrow-hotfix 0.6 pydantic 2.7.0 pydantic_core 2.18.1 pydub 0.25.1 Pygments 2.17.2 pynvml 11.5.0 pyparsing 3.1.2 pytest 8.1.1 python-dateutil 2.9.0.post0 python-multipart 0.0.9 pytz 2024.1 PyYAML 6.0.1 referencing 0.34.0 regex 2024.4.16 requests 2.31.0 responses 0.18.0 rich 13.7.1 rpds-py 0.18.0 ruff 0.4.1 safetensors 0.4.3 scikit-learn 1.4.2 scipy 1.13.0 semantic-version 2.10.0 sentencepiece 0.2.0 setuptools 68.2.2 shellingham 1.5.4 six 1.16.0 sniffio 1.3.1 starlette 0.37.2 sympy 1.12 tensorboard 2.16.2 tensorboard-data-server 0.7.2 threadpoolctl 3.4.0 tokenizers 0.15.2 tomli 2.0.1 tomlkit 0.12.0 toolz 0.12.1 torch 2.1.2 torchaudio 2.1.2 torchdata 0.7.1 torchvision 0.16.2 tqdm 4.66.2 transformers 4.39.0 triton 2.1.0 typer 0.12.3 typing_extensions 4.11.0 tzdata 2024.1 urllib3 2.2.1 uvicorn 0.29.0 websockets 11.0.3 Werkzeug 3.0.2 wheel 0.41.2 xxhash 3.4.1 yarl 1.9.4- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
- 42
- 43
- 44
- 45
- 46
- 47
- 48
- 49
- 50
- 51
- 52
- 53
- 54
- 55
- 56
- 57
- 58
- 59
- 60
- 61
- 62
- 63
- 64
- 65
- 66
- 67
- 68
- 69
- 70
- 71
- 72
- 73
- 74
- 75
- 76
- 77
- 78
- 79
- 80
- 81
- 82
- 83
- 84
- 85
- 86
- 87
- 88
- 89
- 90
- 91
- 92
- 93
- 94
- 95
- 96
- 97
- 98
- 99
- 100
- 101
- 102
- 103
- 104
- 105
- 106
- 107
- 108
- 109
- 110
- 111
- 112
- 113
- 114
- 115
- 116
- 117
- 118
- 119
- 120
- 121
- 122
- 123
- 124
- 125
- 126
- 127
- 128
- 129
- 130
- 131
- 132
- 133
- 134
- 135
- 136
- 137
- 138
- 139
- 140
- 141
- 142
- 143
- 144
- 145
- 146
- 147
- 148
- 149
- 150
- 151
- 152
- 153
- 154
- 155
- 156
- 157
- 158
- 159
- 160
- 161
- 162
- 163
- 164
- 165
- 166
- 167
- 168
- 169
- 170
- 171
- 172
- 173
- 174
- 175
- 176
- 177
- 178
- 179
- 180
- 181
- 182
- 183
- 184
- 185
- 186
- 187
- 188
- 189
- 190
- 191
- 192
- 193
- 194
- 195
- 196
- 197
- 198
- 199
- 200
- 201
- 202
- 203
- 204
- 205
- 206
- 207
- 208
- 209
- 210
- 211
- 212
- 213
- 214
- 215
- 216
- 217
- 218
- 219
- 220
- 221
- 222
- 223
- 224
- 225
- 226
- 227
- 228
- 229
- 230
- 231
- 232
- 233
- 234
- 235
- 236
- 237
- 238
- 239
- 240
- 241
- 242
- 243
- 244
- 245
- 246
- 247
- 248
- 249
- 250
- 251
- 252
- 253
- 254
- 255
- 256
- 257
- 258
- 259
- 260
- 261
- 262
- 263
- 264
- 265
- 266
- 267
- 268
- 269
- 270
- 271
- 272
- 273
- 274
- 275
- 276
- 277
- 278
- 279
- 280
- 281
- 282
- 283
- 284
- 285
- 286
- 287
- 288
- 289
- 290
- 291
- 292
- 293
- 294
- 295
- 296
- 297
- 298
- 299
- 300
- 301
- 302
- 303
- 304
- 305
- 306
- 307
- 308
- 309
- 310
- 311
下载模型：Meta-Llama-3-8B-Instruct
你可以从以下来源下载Meta-Llama-3-8B-Instruct模型。
https://github.com/LlamaFamily/Llama-Chinese?tab=readme-ov-file- 1
(pytorch212) :~$ cd Meta-Llama-3-8B-Instruct (pytorch212) :~/Meta-Llama-3-8B-Instruct$ ls LICENSE generation_config.json model-00004-of-00004.safetensors tokenizer.json README.md model-00001-of-00004.safetensors model.safetensors.index.json tokenizer_config.json USE_POLICY.md model-00002-of-00004.safetensors original config.json model-00003-of-00004.safetensors special_tokens_map.json- 1
- 2
- 3
- 4
- 5
- 6
- 7
使用Meta-Llama-3-8B-Instruct模型进行推理 创建一个名为 QuickStartLlama3.py 的文件，并将以下内容复制到该文件中。
import transformers import torch model_id = "./Meta-Llama-3-8B-Instruct" pipeline = transformers.pipeline( "text-generation", model=model_id, model_kwargs={"torch_dtype": torch.bfloat16}, device="cuda", ) messages = [ {"role": "system", "content": "hello,You are a helpful human assistant!"}, {"role": "user", "content": "介绍一下中国,请用中文回答"}, ] prompt = pipeline.tokenizer.apply_chat_template( messages, tokenize=False, add_generation_prompt=True ) terminators = [ pipeline.tokenizer.eos_token_id, pipeline.tokenizer.convert_tokens_to_ids("<|eot_id|>") ] outputs = pipeline( prompt, max_new_tokens=256, eos_token_id=terminators, do_sample=True, temperature=0.6, top_p=0.9, )- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
执行脚本
(pytorch212) :~$ python QuickStartLlama3.py Loading checkpoint shards: 100%|████████████████████████████| 4/4 [00:02<00:00, 1.35it/s] Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained. Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation. 中国是世界上最 populous 的一个国家，位于亚洲东部，邻近朝鲜、俄罗斯、蒙古、塔吉克斯坦、阿富是世界上第二大经济体，拥有15亿多人口，文化古老，历史悠久，拥有五千多年的文明史。 中国的政治体制是社会主义共和国，首都是北京市，政府是中国共产党领导的。中国的经济体制是社会 中国有着丰富的自然资源，包括煤炭、石油、天然气、铁矿石、铜矿石、铅矿石等。中国的主要工业有非常发达，主要生产粮食、棉花、油料、糖料等农产品。 中国的文化非常多样化，拥有五千多年的文明史，包括儒家、道 (pytorch212) :~$- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
nvidia-smi命令实时查看指定GPU使用情况
watch -n 1 nvidia-smi # 1代表每隔1秒刷新一次GPU使用情况 NVIDIA-SMI 550.76.01 #GRID版本 Driver Version: 552.22 #驱动版本 CUDA Version: 12.4 #CUDA最高支持的版本 GPU：本机中的GPU编号，从0开始，本机只有一块GPU Fan：风扇转速（0%-100%），N/A表示没有风扇 Name：GPU名字/类型，NVIDIA GeForce RTX 3080TI Temp：GPU温度（GPU温度过高会导致GPU频率下降） 68C Perf：性能状态，从P0（最大性能）到P12（最小性能），显示P0，最大性能 Pwr：Usager/Cap：GPU功耗，Usage表示用了多少，Cap表示总共多少 ， 79W / 80W Persistence-M：持续模式状态，持续模式耗能大，为On Bus-Id：GPU总线 00000000:01:00.0 Disp.A：Display Active，表示GPU是否初始化 Off Memory-Usage：显存使用率 15740MiB / 16384MiB，表示已接近占满 Volatile GPU-UTil：GPU使用率，87% Uncorr. ECC：是否开启错误检查和纠错技术，0/DISABLED，1/ENABLED，为N/A Compute M：计算模式，0/DEFAULT，1/EXCLUSIVE_PROCESS，2/PROHIBITED，为Default Processes：显示每个进程占用的显存使用率、进程号、占用的哪个GPU，/python3.10 GPU Memory Usage #该进程占用的显存。- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
2（最小性能），显示P0，最大性能
Pwr：Usager/Cap：GPU功耗，Usage表示用了多少，Cap表示总共多少 ， 79W / 80W
Persistence-M：持续模式状态，持续模式耗能大，为On
Bus-Id：GPU总线 00000000:01:00.0
Disp.A：Display Active，表示GPU是否初始化 Off
Memory-Usage：显存使用率 15740MiB / 16384MiB，表示已接近占满
Volatile GPU-UTil：GPU使用率，87%
Uncorr. ECC：是否开启错误检查和纠错技术，0/DISABLED，1/ENABLED，为N/A
Compute M：计算模式，0/DEFAULT，1/EXCLUSIVE_PROCESS，2/PROHIBITED，为Default
Processes：显示每个进程占用的显存使用率、进程号、占用的哪个GPU，/python3.10
GPU Memory Usage #该进程占用的显存。
``
-
相关阅读:
数据结构 - 链表
g++安装 yum -y install gcc+ gcc-c++ 报错Unable to find a match: gcc+
【文献阅读】【NC 2023】基于分子组装的可解释深度学习方法实现精准逆合成预测以及路径规划
ICO操作Bean管理的（bean的作用域和生命周期）
使用Vue的transition组件写一个数字滚动竟然如此简单
[补题记录] Atcoder Beginner Contest 323（E、F）
51单片机学习笔记_1单片机介绍，C语言基础语法
什么是分布式锁，解释分布式锁的实现方式和常见的分布式锁算法
【微机原理｜课程报告】计算器课设报告
猿创征文 |【C++】动态内存的分配与回收
- 原文地址：https://blog.csdn.net/u012339179/article/details/138168826
