1.Windows 环境下 llama.cpp 编译 + Qwen 模型本地部署全指南2.本地部署千问 2.5-1.5B-GGUF + LangChain 封装学习3.Windows平台下CUDA安装及llama.cpp使用教程4.Windows 10 下 AMD RX580 调用 Ollama GPU 加速教程5.Windows 环境下 llama.cpp 编译运行指南6.LangChain+Llama.cpp 本地模型与Agent工具调用7.LangGraph 会话切换与短期记忆管理8.LangChain+FastMCP 搭建大模型工具调用服务9.LangGraph+PostgreSQL 会话记忆持久化存储10.LangChain 实现NaiveRAG朴素向量检索生成11.DeepSeek Harness本地部署教程｜搭配LlamaCpp本地模型全程零基础搭建12.LangChain 实现AdvancedRAG增强向量检索生成13.LangGraph Server Agent 框架本地部署指南14.Python 原生封装 Llama.cpp 大模型推理接口15.LangChain 消息流输出与结构化处理16.Ubuntu 大模型HF转GGUF全流程实践指南17.轻量化小模型MiniMind从训练到落地指南18.Ubuntu 本地部署Ollama+OpenWebUI教程19.千问大模型二次LoRA‑SFT指令微调指南20.千问大模型完整RLHF全参数微调指南
21.LLama-Factory 实现大模型LoRA-SFT微调指南
22.基于 vLLM+Nginx 构建负载均衡推理集群23.基于 HuggingFace Tokenizers 训练自定义分词器24.基于魔搭 MS-Swift 实现大模型微调落地指南25.基于 LlamaIndex 实现RAG向量检索入门26.基于 LlamaIndex+PostgreSQL 实现RAG持久化27.LangGraph 基于状态图构建工作流入门28.基于 LlamaIndex 实现 Hybrid‑RAG 召回指南29.MysticMirror 自研大模型全维度自动化测试套件（42 项用例）LLaMA Factory 是一款开源、低代码、一站式大语言与多模态模型微调框架，用于降低大模型的微调落地门槛。框架兼容 Qwen、LLaMA、ChatGLM、LLaVA 等上百款主流开源模型，支持增量预训练、SFT 监督微调、DPO、KTO、ORPO 等多种训练对齐方案，原生集成 LoRA、QLoRA 量化微调技术，可在消费级显卡完成轻量级模型微调。框架支持可视化网页端和命令行双操作模式，完整覆盖模型训练、评估、权重合并、推理部署全流程，适配科研实验、行业轻量化模型定制等场景。
本次实践基于 Ubuntu 22.04 系统、RTX 4090 24G 显卡、CUDA12.8 环境，全程使用国内镜像加速，解决外网下载慢、超时问题。
安装与配置
1、备份原有源文件，替换为阿里云镜像源，提升系统包下载速度。
root@localhost:~# cp /etc/apt/sources.list /etc/apt/sources.list.bak
root@localhost:~# bash -c 'cat > /etc/apt/sources.list <
deb http://mirrors.aliyun.com/ubuntu/ jammy main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ jammy-security main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ jammy-updates main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ jammy-backports main restricted universe multiverse
EOF'
root@localhost:~# apt update
2、安装显卡检测工具，校验 NVIDIA 显卡识别状态，确保硬件环境正常。
root@localhost:~# apt install -y pciutils kmod
root@localhost:~# lspci | grep -i nvidia
01:00.0 VGA compatible controller: NVIDIA Corporation Device 2684 (rev a1)
01:00.1 Audio device: NVIDIA Corporation Device 22ba (rev a1)
25:00.0 VGA compatible controller: NVIDIA Corporation Device 2684 (rev a1)
25:00.1 Audio device: NVIDIA Corporation Device 22ba (rev a1)
41:00.0 VGA compatible controller: NVIDIA Corporation Device 2684 (rev a1)
41:00.1 Audio device: NVIDIA Corporation Device 22ba (rev a1)
61:00.0 VGA compatible controller: NVIDIA Corporation Device 2684 (rev a1)
61:00.1 Audio device: NVIDIA Corporation Device 22ba (rev a1)
81:00.0 VGA compatible controller: NVIDIA Corporation Device 2684 (rev a1)
81:00.1 Audio device: NVIDIA Corporation Device 22ba (rev a1)
a1:00.0 VGA compatible controller: NVIDIA Corporation Device 2684 (rev a1)
a1:00.1 Audio device: NVIDIA Corporation Device 22ba (rev a1)
c1:00.0 VGA compatible controller: NVIDIA Corporation Device 2684 (rev a1)
c1:00.1 Audio device: NVIDIA Corporation Device 22ba (rev a1)
e1:00.0 VGA compatible controller: NVIDIA Corporation Device 2684 (rev a1)
e1:00.1 Audio device: NVIDIA Corporation Device 22ba (rev a1)
root@localhost:~# nvidia-smi
Wed Sep 9 14:28:46 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.86.10 Driver Version: 570.86.10 CUDA Version: 12.8 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GeForce RTX 4090 On | 00000000:C1:00.0 Off | Off |
| 32% 31C P8 22W / 405W | 1MiB / 24564MiB | 0% Default |
| | | N/A |
+-----------------------------------------------------------------------------------------+
root@localhost:~# ls /dev/nvidia*
/dev/nvidia-uvm /dev/nvidia-uvm-tools /dev/nvidia5 /dev/nvidiactl
/dev/nvidia-caps:
nvidia-cap1 nvidia-cap2
3、若系统未安装显卡驱动，执行以下命令安装官方推荐驱动，并禁用开源 nouveau 驱动。
禁用系统默认驱动
root@localhost:~# tee /etc/modprobe.d/blacklist-nouveau.conf <<'EOF'
blacklist nouveau
options nouveau modeset=0
EOF
添加显卡驱动PPA源，并安装 NVIDIA 驱动
root@localhost:~# apt install -y alsa-utils software-properties-common
root@localhost:~# add-apt-repository ppa:graphics-drivers/ppa
root@localhost:~# apt upgrade -y
# 查看推荐驱动
root@localhost:~# ubuntu-drivers devices
# 手动指定版本 recommended 标签则为推荐版本
root@localhost:~# sudo apt install -y nvidia-driver-570
# 重启验证显卡驱动
root@localhost:~# sudo reboot
4、搭建独立虚拟环境，避免依赖版本冲突。
root@localhost:~# apt install -y software-properties-common
root@localhost:~# add-apt-repository -y ppa:deadsnakes/ppa
root@localhost:~# apt install -y python3.12 python3.12-venv python3.12-dev git vim
root@localhost:~# python3.12 -m venv myvenv
root@localhost:~# source myvenv/bin/activate
5、根据显卡 CUDA 版本适配 PyTorch 版本，本次 CUDA12.8 对应安装 Torch2.8.0，使用清华、上交双镜像加速。
其他的版本对应预览表：
| CUDA 编译包 | Linux 最低驱动 | Windows 最低驱动 | PyTorch 版本 |
|---|---|---|---|
| CUDA 13.0 | ≥ 580.30.02 | ≥ 581.06 | 2.9 / 2.12 |
| CUDA 12.9 | ≥ 575.51.03 | ≥ 576.02 | 2.8 |
| CUDA 12.8 | ≥ 570.26 | ≥ 570.65 | 2.7/ 2.8 / 2.9 / 2.10 / 2.11 |
| CUDA 12.6 | ≥ 560.28.03 | ≥ 561.17 | 2.6 / 2.7 ~ 2.12 |
| CUDA 12.4 | ≥ 550.54.14 | ≥ 551.23 | 2.5 / 2.6 |
| CUDA 12.1 | ≥ 525.60.13 | ≥ 527.41 | 2.2 / 2.3 / 2.4 |
| CUDA 11.8 | ≥ 450.80.02 | ≥ 452.39 | 2.0 ~ 2.7 |
依次执行命令安装，该过程较慢
root@localhost:~# pip install torch==2.8.0 torchvision==0.23.0 torchaudio==2.8.0 \
-i https://pypi.tuna.tsinghua.edu.cn/simple \
--extra-index-url https://mirror.sjtu.edu.cn/pytorch-wheels/cu128
root@localhost:~# pip install transformers==4.48.2 accelerate -i https://pypi.tuna.tsinghua.edu.cn/simple
root@localhost:~# pip list
Package Version
------------------------ ------------
accelerate 1.15.0
certifi 2026.7.22
charset-normalizer 3.5.1
filelock 3.32.6
fsspec 2026.7.0
hf-xet 1.6.0
huggingface_hub 0.36.2
idna 3.19
Jinja2 3.1.6
MarkupSafe 3.0.3
mpmath 1.3.0
networkx 3.6.1
numpy 2.5.3
nvidia-cublas-cu12 12.8.4.1
nvidia-cuda-cupti-cu12 12.8.90
nvidia-cuda-nvrtc-cu12 12.8.93
nvidia-cuda-runtime-cu12 12.8.90
nvidia-cudnn-cu12 9.10.2.21
nvidia-cufft-cu12 11.3.3.83
nvidia-cufile-cu12 1.13.1.3
nvidia-curand-cu12 10.3.9.90
nvidia-cusolver-cu12 11.7.3.90
nvidia-cusparse-cu12 12.5.8.93
nvidia-cusparselt-cu12 0.7.1
nvidia-nccl-cu12 2.27.3
nvidia-nvjitlink-cu12 12.8.93
nvidia-nvtx-cu12 12.8.90
packaging 26.3
pillow 12.3.0
pip 25.0.1
psutil 7.2.2
PyYAML 6.0.3
regex 2026.9.3
requests 2.34.2
safetensors 0.8.0
setuptools 84.0.0
sympy 1.14.0
tokenizers 0.21.4
torch 2.8.0+cu128
torchaudio 2.8.0+cu128
torchvision 0.23.0+cu128
tqdm 4.70.0
transformers 4.48.2
triton 3.4.0
typing_extensions 4.16.0
urllib3 2.7.0
6、执行 Python 代码校验 PyTorch、CUDA、显卡识别状态。
import torch
print("torch版本:", torch.__version__)
print("torch编译用的CUDA版本:", torch.version.cuda)
print("CUDA是否可用:", torch.cuda.is_available())
print("GPU数量:", torch.cuda.device_count())
if torch.cuda.is_available():
print("GPU名称:", torch.cuda.get_device_name(0))
# -----------------------------------------
torch版本: 2.8.0+cu128
torch编译用的CUDA版本: 12.8
CUDA是否可用: True
GPU数量: 1
GPU名称: NVIDIA GeForce RTX 4090
7、编译 Llama-Factory 镜像并安装。
root@localhost:~# git clone https://gitee.com/lyshark/LLaMA-Factory.git
root@localhost:~# cd LLaMA-Factory
root@localhost:~# pip install -i https://pypi.tuna.tsinghua.edu.cn/simple -e .
root@localhost:~# pip list
Package Version
------------------------ ------------
accelerate 1.11.0
aiofiles 24.1.0
aiohappyeyeballs 2.7.1
aiohttp 3.14.3
aiosignal 1.4.0
annotated-doc 0.0.5
annotated-types 0.8.0
antlr4-python3-runtime 4.9.3
anyio 4.15.1
attrs 26.1.0
av 16.0.0
brotli 1.2.0
certifi 2026.7.22
cffi 2.1.1
charset-normalizer 3.5.1
click 8.5.0
contourpy 1.3.3
cryptography 50.0.1
cycler 0.12.1
datasets 4.0.0
dill 0.3.8
docstring_parser 0.18.0
einops 0.8.2
fastapi 0.141.1
ffmpy 1.0.0
filelock 3.32.6
fire 0.7.1
fonttools 4.64.0
frozenlist 1.8.0
fsspec 2025.3.0
gradio 5.50.0
gradio_client 1.14.0
groovy 0.1.2
h11 0.16.0
hf_transfer 0.1.9
hf-xet 1.6.0
httpcore 1.0.9
httpx 0.28.1
huggingface_hub 1.30.0
idna 3.19
Jinja2 3.1.6
kiwisolver 1.5.1
llamafactory 0.9.6.dev0
markdown-it-py 4.2.0
MarkupSafe 3.0.3
matplotlib 3.11.1
mdurl 0.1.2
modelscope 1.40.0
modelscope-hub 0.4.1
mpmath 1.3.0
multidict 6.8.0
multiprocess 0.70.16
networkx 3.6.1
numpy 2.5.3
nvidia-cublas-cu12 12.8.4.1
nvidia-cuda-cupti-cu12 12.8.90
nvidia-cuda-nvrtc-cu12 12.8.93
nvidia-cuda-runtime-cu12 12.8.90
nvidia-cudnn-cu12 9.10.2.21
nvidia-cufft-cu12 11.3.3.83
nvidia-cufile-cu12 1.13.1.3
nvidia-curand-cu12 10.3.9.90
nvidia-cusolver-cu12 11.7.3.90
nvidia-cusparse-cu12 12.5.8.93
nvidia-cusparselt-cu12 0.7.1
nvidia-nccl-cu12 2.27.3
nvidia-nvjitlink-cu12 12.8.93
nvidia-nvtx-cu12 12.8.90
omegaconf 2.3.1
orjson 3.12.0
packaging 26.3
pandas 2.3.3
peft 0.18.1
pillow 11.3.0
pip 25.0.1
propcache 0.5.2
protobuf 7.36.1
psutil 7.2.2
pyarrow 25.0.1
pycparser 3.0
pydantic 2.12.3
pydantic_core 2.41.4
pydub 0.25.1
Pygments 2.21.0
pyparsing 3.3.2
python-dateutil 2.9.0.post0
python-multipart 0.0.32
pytz 2026.3.post1
PyYAML 6.0.3
regex 2026.9.3
requests 2.34.2
rich 15.0.0
ruff 0.16.6
safehttpx 0.1.7
safetensors 0.8.0
scipy 1.18.1
semantic-version 2.10.0
sentencepiece 0.2.2
setuptools 84.0.0
shellingham 1.5.4
shtab 1.12.1
six 1.17.0
sse-starlette 3.4.11
starlette 0.52.1
sympy 1.14.0
termcolor 3.3.0
tiktoken 0.14.0
tokenizers 0.22.2
tomlkit 0.13.3
torch 2.8.0+cu128
torchaudio 2.8.0+cu128
torchdata 0.11.0
torchvision 0.23.0+cu128
tqdm 4.70.0
transformers 5.8.0
triton 3.4.0
trl 0.24.0
typer 0.27.2
typing_extensions 4.16.0
typing-inspection 0.4.4
tyro 0.8.14
tzdata 2026.3
urllib3 2.7.0
uvicorn 0.52.4
websockets 15.0.1
xxhash 4.0.1
yarl 1.24.5
root@localhost:~# llamafactory-cli version
----------------------------------------------------------
| Welcome to LLaMA Factory, version 0.9.6.dev0 |
| |
| Project page: https://github.com/hiyouga/LLaMA-Factory |
----------------------------------------------------------
监督微调
本次微调采用问答数据集，原始数据为 jsonl 格式，需转换为 LlamaFactory 标准的 Alpaca 训练格式。
1、通过 ModelScope 下载 Qwen3.5-0.8B-Instruct 轻量化对话模型，适合消费级显卡微调。
root@localhost:~/# mkdir /data
root@localhost:~/# cd /data
root@localhost:~/# modelscope download --model icyfenix/Qwen3.5-0.8B-Instruct --local_dir /data/qwen3.5\-0.8B\-Instruct
root@localhost:~/# mv qwen3.5‑0.8B‑Instruct qwen3.5
2、准备训练材料，数据包含 question、answer 字段，文件名称叫做train.json，放入到LlamaFactory/data 目录下，以下结构是标准训练集结构。
[
{
"instruction": "你是一个助手",
"input": "用户问题",
"output": "回答"
},
{
"instruction": "你是一个助手",
"input": "第二个问题",
"output": "对应的回答"
}
]
3、继续使用《千问大模型完整RLHF全参数微调指南》文章中的医疗数据集，并将其做清洗处理。
下载数据集
root@localhost:~/qwen# wget https://modelscope.cn/datasets/krisfu/delicate_medical_r1_data/resolve/master/r1_data_example.jsonl
root@localhost:~/qwen# ls -lh
直接使用脚本将其转换为符合规范的格式，读取 r1_data_example.jsonl 文件，把每条的 question → input、answer → output 进行关联，并固定 instruction 为特定提示词，输出标准 json 数组格式。
import json
def convert_jsonl_to_json(jsonl_file_path, out_json_path):
output_data = []
with open(jsonl_file_path, "r", encoding="utf-8") as f:
for line in f:
line = line.strip()
if not line:
continue
item = json.loads(line)
new_sample = {
"instruction": "你是一个医疗问答助手，请规范回答用户提问。",
"input": item["question"],
"output": item["answer"]
}
output_data.append(new_sample)
with open(out_json_path, "w", encoding="utf-8") as fw:
json.dump(output_data, fw, ensure_ascii=False, indent=4)
print(f"转换完成，输出文件: {out_json_path}")
if __name__ == "__main__":
jsonl_path = "/data/r1_data_example.jsonl"
save_json_path = "/data/train.json"
convert_jsonl_to_json(jsonl_path, save_json_path)
输出成train.json文件，并放入到/data目录下
root@localhost:~/# cd /data/
root@localhost:~/qwen# ls -lh
total 12M
drwxr-xr-x 2 root root 4.0K Sep 9 04:32 qwen3.5
-rw-r--r-- 1 root root 8.8M Apr 22 2025 r1_data_example.jsonl
-rw-r--r-- 1 root root 2.4M Sep 9 04:47 train.json
4、覆盖写入自定义数据集，修改 LlamaFactory/data/dataset_info.json 直接覆盖。
"mydata"就是数据集名称
root@localhost:~/# cat <
{
"mydata": {
"file_name": "/data/train.json",
"format": "alpaca"
}
}
EOF
5、在 LlamaFactory 根目录新建 sft.yaml，配置 LoRA 微调核心参数。
root@localhost:~/# pip install tiktoken
root@localhost:~/# cat <
model_name_or_path: /data/qwen3.5
dataset: mydata
template: qwen
finetuning_type: lora
lora_target: all
lora_rank: 8
lora_alpha: 16
lora_dropout: 0.05
stage: sft
do_train: true
num_train_epochs: 3
per_device_train_batch_size: 4
gradient_accumulation_steps: 2
learning_rate: 5e-5
lr_scheduler_type: cosine
warmup_steps: 50
weight_decay: 0.01
dataloader_num_workers: 0
output_dir: /data/qwen3.5_sft
save_steps: 100
logging_steps: 10
overwrite_output_dir: true
gradient_checkpointing: true
fp16: true
EOF
6、在 LLaMA-Factory 目录下执行命令启动训练。
root@localhost:/data# mkdir /data/qwen3.5_sft
root@localhost:/data# mkdir /data/qwen3.5_lora_merged
root@localhost:~/LLaMA-Factory# llamafactory-cli train sft.yaml
{'train_runtime': '21.48', 'train_samples_per_second': '2.793', 'train_steps_per_second': '0.419', 'train_loss': '2.202', 'epoch': '3'}
100%|███████████████████████████████████████████████████| 9/9 [00:21<00:00, 2.39s/it]
root@localhost:/data# ls -lh
total 49M
drwxr-xr-x 2 root root 4.0K Sep 9 07:40 checkpoint-9
drwxr-xr-x 2 root root 4.0K Sep 9 07:26 qwen3.5
drwxr-xr-x 2 root root 10 Sep 9 07:44 qwen3.5_lora_merged
drwxr-xr-x 3 root root 4.0K Sep 9 07:42 qwen3.5_sft
-rw-r--r-- 1 root root 8.8M Apr 27 00:54 r1_data_example.jsonl
-rw-r--r-- 1 root root 19K Sep 9 07:38 train.json
7、训练结束后通过执行 merge_lora.yaml 脚本实现合并模型权重。
root@localhost:~/LLaMA-Factory# cat <
model_name_or_path: /data/qwen3.5
adapter_name_or_path: /data/qwen3.5_sft
template: qwen
finetuning_type: lora
export_dir: /data/qwen3.5_lora_merged
export_legacy_format: false
EOF
root@localhost:~/LLaMA-Factory# llamafactory-cli export merge_lora.yaml
Loading weights: 100%|███████████████████████████████████████████| 473/473 [00:00<00:00, 6414.93it/s]
root@localhost:/data# ls -lh
total 49M
drwxr-xr-x 2 root root 4.0K Sep 9 07:26 qwen3.5
drwxr-xr-x 2 root root 10 Sep 9 07:44 qwen3.5_lora_merged
drwxr-xr-x 3 root root 4.0K Sep 9 07:42 qwen3.5_sft
-rw-r--r-- 1 root root 8.8M Apr 27 00:54 r1_data_example.jsonl
-rw-r--r-- 1 root root 19K Sep 9 07:38 train.json
root@localhost:/data/qwen3.5_lora_merged# ls -lh
total 1.7G
-rw-r--r-- 1 root root 464 Sep 9 07:46 Modelfile
-rw-r--r-- 1 root root 7.6K Sep 9 07:46 chat_template.jinja
-rw-r--r-- 1 root root 2.7K Sep 9 07:46 config.json
-rw-r--r-- 1 root root 115 Sep 9 07:46 generation_config.json
-rw------- 1 root root 1.6G Sep 9 07:46 model.safetensors
-rw-r--r-- 1 root root 1.2K Sep 9 07:46 processor_config.json
-rw-r--r-- 1 root root 20M Sep 9 07:46 tokenizer.json
-rw-r--r-- 1 root root 1.2K Sep 9 07:46 tokenizer_config.json
模型测试
编写推理配置文件，启动命令行交互式对话，测试医疗微调效果。
1、新建 infer_lora.yaml 放在 LLaMA-Factory 根目录
root@localhost:~/LLaMA-Factory# cat <
model_name_or_path: /data/qwen3.5_lora_merged
template: qwen
temperature: 0.4
top_p: 0.8
max_new_tokens: 512
EOF
2、启动终端对话
root@localhost:~/LLaMA-Factory# llamafactory-cli chat infer_lora.yaml
Loading weights: 100%|███████████████████████████████████| 473/473 [00:00<00:00, 1092.90it/s]
Welcome to the CLI application, use `clear` to remove the history, use `exit` to exit the application.
User: 你好
Assistant: <think>
用户打招呼，这是一个简单的问候。我应该用友好的方式回应，保持亲切和友好的语气。
</think>
你好！很高兴见到你。有什么我可以帮你的吗？