Git 服务器上的 LFS 下载

以llama为例：
https://huggingface.co/meta-llama/Llama-2-7b-hf

Github

# 1. 安装完成后，首先先初始化；如果有反馈，一般表示初始化成功
git lfs install
​
# 2. 如果刚刚下载的那个项目没啥更改，重新下一遍，不算麻烦事（因为下载大文件，一般会比较慢）
git lfs clone https://huggingface.co/meta-llama/Llama-2-7b-hf
# 在下载的过程中，你也可以查看一下，你刚刚无法解析的那个pkl大文件，是不是在这个项目中，(进入项目目录)使用如下指令：
cd Llama-2-7b-hf/

git lfs track
​
# 3. 如果不想重新下载整个项目，可以使用如下命令，单独下载需要使用lfs下载的大文件。
git lfs fetch
git lfs checkout
#（备选：git lfs pull），不建议
1
2
3
4
5
6
7
8
9
10
11
12
13
14

GIT_LFS_SKIP_SMUDGE=0 git clone https://huggingface.co/meta-llama/Llama-2-7b-hf
1

报错

$ git clone https://huggingface.co/meta-llama/Llama-2-7b-hf
Cloning into 'Llama-2-7b-hf'...
fatal: unable to access 'https://huggingface.co/meta-llama/Llama-2-7b-hf/': Failed to connect to huggingface.co port 443 after 21050 ms: Timed out
1
2
3

找国内镜像

GIT_LFS_SKIP_SMUDGE=0

rm -rf *
1

git lfs clone clone https://gitee.com/hf-models/Llama-2-7b-hf.git
GIT_LFS_SKIP_SMUDGE=1 git clone https://gitee.com/hf-models/Llama-2-7b-hf.git
GIT_LFS_SKIP_SMUDGE=0 git clone https://gitee.com/hf-models/Llama-2-7b-hf.git
1
2
3

1

报错

$ git lfs clone https://gitee.com/hf-models/Llama-2-7b-hf.git
WARNING: `git lfs clone` is deprecated and will not be updated
          with new flags from `git clone`

`git clone` has been updated in upstream Git to have comparable
speeds to `git lfs clone`.
Cloning into 'Llama-2-7b-hf'...
remote: Enumerating objects: 33, done.
remote: Counting objects: 100% (33/33), done.
remote: Compressing objects: 100% (32/32), done.
remote: Total 33 (delta 9), reused 0 (delta 0), pack-reused 0
Receiving objects: 100% (33/33), 491.84 KiB | 631.00 KiB/s, done.
Resolving deltas: 100% (9/9), done.
Not in a Git repository.
1
2
3
4
5
6
7
8
9
10
11
12
13
14

注意 Not in a Git repository.报错，查看：

$ git rev-parse --is-inside-work-tree
fatal: unsafe repository ('E:/tst/t/Llama-2-7b-hf' is owned by someone else)
To add an exception for this directory, call:
        git config --global --add safe.directory E:/tst/t/Llama-2-7b-hf
1
2
3
4

因此：

git config --global --add safe.directory E:/tst/t/Llama-2-7b-hf

git lfs track
# 3. 如果不想重新下载整个项目，可以使用如下命令，单独下载需要使用lfs下载的大文件。
git lfs fetch
git lfs checkout
#（备选：git lfs pull），不建议
1
2
3
4
5
6
7

使用git lfs fetch，报错：

$ git lfs fetch
fetch: Fetching reference refs/heads/main
[4ec71fd53e99766de38f24753b30c9e8942630e9e576a1ba27b0ec531e87be41] Object not found: [404] Object not found
[60632e7f2530e713d57cfe2152019a01b1018070ce497e1f0c7fae4bf1340314] Object not found: [404] Object not found
[fa574ee4fa536cc1dfc1c1baf622144fa037a5259967112eeb1a40c7cc00484a] Object not found: [404] Object not found
[fd491dd3738e74dd4e18b964b9f8a15f785d487ab2d0c03c7a7b22fdac7bce1a] Object not found: [404] Object not found
[41780b5dac322ac35598737e99208d90bdc632a1ba3389ebedbb46a1d8385a7f] Object not found: [404] Object not found
error: failed to fetch some objects from 'https://gitee.com/hf-models/Llama-2-7b-hf.git/info/lfs'
1
2
3
4
5
6
7
8

原因：

• 对象不存在：指定的对象在远程仓库中不存在。这可能是因为对象已被删除、重命名或移动。请确保你正在尝试获取正确的对象。
• 访问权限限制：你可能没有足够的权限来访问远程仓库中的对象。请确保你具有正确的访问权限，并且已经通过身份验证进行访问。

2

报错：

$ GIT_LFS_SKIP_SMUDGE=0 git clone https://gitee.com/hf-models/Llama-2-7b-hf.git
Cloning into 'Llama-2-7b-hf'...
remote: Enumerating objects: 33, done.
remote: Counting objects: 100% (33/33), done.
remote: Compressing objects: 100% (32/32), done.
remote: Total 33 (delta 9), reused 0 (delta 0), pack-reused 0
Receiving objects: 100% (33/33), 491.84 KiB | 729.00 KiB/s, done.
Resolving deltas: 100% (9/9), done.
Updating files: 100% (18/18), done.
Downloading model-00001-of-00002.safetensors (10 GB)/s
Error downloading object: model-00001-of-00002.safetensors (4ec71fd): Smudge error: Error downloading model-00001-of-00002.safetensors (4ec71fd53e99766de38f24753b30c9e8942630e9e576a1ba27b0ec531e87be41): [4ec71fd53e99766de38f24753b30c9e8942630e9e576a1ba27b0ec531e87be41] Object not found: [404] Object not found

Errors logged to 'E:\Llama-2-7b-hf\.git\lfs\logs\20231112T152236.9629598.log'.
Use `git lfs logs last` to view the log.
error: external filter 'git-lfs filter-process' failed
fatal: model-00001-of-00002.safetensors: smudge filter lfs failed
warning: Clone succeeded, but checkout failed.
You can inspect what was checked out with 'git status'
and retry with 'git restore --source=HEAD :/'
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

https://gitee.com/hf-models/Llama-2-7b-hf/tree/main

ref

https://blog.shipengx.com/archives/af0e9080.html

https://zhuanlan.zhihu.com/p/350011379