Github每日精选（第41期）：跨端数据同步加密工具restic

restic

restic 是一个快速、安全、高效的备份工具，有的时候我们在做服务端的时候，有些文件我们需要进行另外的保存，可能是因为这是重要的文件需要进行安全的备份，也可能是因为文件的私密性。

在安全方面上restic 做得不错，但是他就失去了像网盘那样的便利性，这也是restic 为啥只在一部分人里面流行而已。

github的地址在这里。

restic 有几个优势：

• 密码登陆和数据加密
• 版本管理，像git一样，他的仓库可以还原到最初的版本，我们需要哪个版本都可以通过像git那样还原回来这就最大的方便了。
• 开源免费，跨平台可用。

安装

在 Debian 上，可以从官方仓库安装的软件包restic ，apt-get：

$ apt-get install restic
1

使用

在下面的使用过程中，如果你使用过git的代码管理的话，你会发现，restic 跟git很像，甚至是一样的，一样的命令。

创建仓库

初始化一个仓库，按提示输入密码，这个密码打死也不能告诉别人，但是你也不能自己忘掉，忘掉的话这个仓库基本是毁了。

restic init --repo /tmp/backup
1

提示我们输入两次代码：

enter password for new repository: 
enter password again: 
created restic repository 25d29858c9 at /tmp/backup

Please note that knowledge of your password is required to access
the repository. Losing your password means that your data is
irrecoverably lost.
1
2
3
4
5
6
7

告诉我们，这个密码很重要，不要丢了。

备份文件

这样我们就有一个仓库/tmp/backup了，有了仓库以后，我们就可以把想备份的文件放到这个仓库来了。

restic --repo /tmp/backup backup manim/
1

这里我们要备份当前文档下的manim 文件夹。提示我们输入密码：

enter password for repository: 
repository 25d29858 opened successfully, password is correct
created new cache in /root/.cache/restic
no parent snapshot found, will read all files

Files:         207 new,     0 changed,     0 unmodified
Dirs:           57 new,     0 changed,     0 unmodified
Added to the repo: 75.624 MiB

processed 207 files, 75.530 MiB in 0:01
snapshot 5cc831ec saved
1
2
3
4
5
6
7
8
9
10
11

简单的列出了我们这个文件夹里的相关信息，还有我们备份的版本号为5cc831ec 。

我们在备份另外一个压缩文件夹。

restic --repo /tmp/backup backup gotty_linux_amd64.tar.gz
1

显示的信息如下：

enter password for repository: 
repository 25d29858 opened successfully, password is correct
no parent snapshot found, will read all files

Files:           1 new,     0 changed,     0 unmodified
Dirs:            0 new,     0 changed,     0 unmodified
Added to the repo: 2.815 MiB

processed 1 files, 2.814 MiB in 0:00
snapshot 37349667 saved
1
2
3
4
5
6
7
8
9
10

查看仓库版本

有了多次的备份以后，生成了不同的版本号，之前我们备份了两次，来看看这个两次的版本号：

restic -r /tmp/backup snapshots
1

显示的信息如下：

enter password for repository: 
repository 25d29858 opened successfully, password is correct
ID        Time                 Host        Tags        Paths
-----------------------------------------------------------------------------------------
5cc831ec  2022-08-16 16:41:16  lqp                     /home/lqp/manim
37349667  2022-08-16 16:44:07  lqp                     /home/lqp/gotty_linux_amd64.tar.gz
-----------------------------------------------------------------------------------------
2 snapshots
1
2
3
4
5
6
7
8

版本还原

要恢复数据出来使用 restore 命令：

restic -r /tmp/backup restore 37349667 --target resticback
1

这样就把版本号37349667 恢复到resticback 的文件夹了。

enter password for repository: 
repository 25d29858 opened successfully, password is correct
restoring <Snapshot 37349667 of [/home/lqp/gotty_linux_amd64.tar.gz] at 2022-08-16 16:44:07.066697148 +0800 CST by root@lqp> to resticback
root@lqp:/home/lqp# cd resticback/
1
2
3
4

SFTP 备份还原

本地备份可能还不过瘾，来个 SFTP 备份，为了通过 SFTP 备份数据，您必须首先使用 SSH 设置服务器并让它知道您的公钥。无密码登录很重要，因为如果服务器提示输入凭据，则无法进行自动备份。

配置服务器后，可以通过更改init命令中的 URL 方案来简单地设置 SFTP 存储库：

$ restic -r sftp:user@host:/srv/restic-repo init
enter password for new repository:
enter password again:
created restic repository f1c6108821 at sftp:user@host:/srv/restic-repo
Please note that knowledge of your password is required to access the repository.
Losing your password means that your data is irrecoverably lost.
1
2
3
4
5
6

源码分析

restic采用的是纯go语言编写，对于go语言比较感兴趣的，还有想了解Snapshot原理的可以参考：

package restic

import (
	"context"
	"fmt"
	"os/user"
	"path/filepath"
	"sync"
	"time"

	"golang.org/x/sync/errgroup"

	"github.com/restic/restic/internal/debug"
)

// Snapshot is the state of a resource at one point in time.
type Snapshot struct {
	Time     time.Time `json:"time"`
	Parent   *ID       `json:"parent,omitempty"`
	Tree     *ID       `json:"tree"`
	Paths    []string  `json:"paths"`
	Hostname string    `json:"hostname,omitempty"`
	Username string    `json:"username,omitempty"`
	UID      uint32    `json:"uid,omitempty"`
	GID      uint32    `json:"gid,omitempty"`
	Excludes []string  `json:"excludes,omitempty"`
	Tags     []string  `json:"tags,omitempty"`
	Original *ID       `json:"original,omitempty"`

	id *ID // plaintext ID, used during restore
}

// NewSnapshot returns an initialized snapshot struct for the current user and
// time.
func NewSnapshot(paths []string, tags []string, hostname string, time time.Time) (*Snapshot, error) {
	absPaths := make([]string, 0, len(paths))
	for _, path := range paths {
		p, err := filepath.Abs(path)
		if err == nil {
			absPaths = append(absPaths, p)
		} else {
			absPaths = append(absPaths, path)
		}
	}

	sn := &Snapshot{
		Paths:    absPaths,
		Time:     time,
		Tags:     tags,
		Hostname: hostname,
	}

	err := sn.fillUserInfo()
	if err != nil {
		return nil, err
	}

	return sn, nil
}

// LoadSnapshot loads the snapshot with the id and returns it.
func LoadSnapshot(ctx context.Context, loader LoaderUnpacked, id ID) (*Snapshot, error) {
	sn := &Snapshot{id: &id}
	err := LoadJSONUnpacked(ctx, loader, SnapshotFile, id, sn)
	if err != nil {
		return nil, err
	}

	return sn, nil
}

// SaveSnapshot saves the snapshot sn and returns its ID.
func SaveSnapshot(ctx context.Context, repo SaverUnpacked, sn *Snapshot) (ID, error) {
	return SaveJSONUnpacked(ctx, repo, SnapshotFile, sn)
}

// ForAllSnapshots reads all snapshots in parallel and calls the
// given function. It is guaranteed that the function is not run concurrently.
// If the called function returns an error, this function is cancelled and
// also returns this error.
// If a snapshot ID is in excludeIDs, it will be ignored.
func ForAllSnapshots(ctx context.Context, be Lister, loader LoaderUnpacked, excludeIDs IDSet, fn func(ID, *Snapshot, error) error) error {
	var m sync.Mutex

	// track spawned goroutines using wg, create a new context which is
	// cancelled as soon as an error occurs.
	wg, ctx := errgroup.WithContext(ctx)

	ch := make(chan ID)

	// send list of snapshot files through ch, which is closed afterwards
	wg.Go(func() error {
		defer close(ch)
		return be.List(ctx, SnapshotFile, func(fi FileInfo) error {
			id, err := ParseID(fi.Name)
			if err != nil {
				debug.Log("unable to parse %v as an ID", fi.Name)
				return nil
			}

			if excludeIDs.Has(id) {
				return nil
			}

			select {
			case <-ctx.Done():
				return nil
			case ch <- id:
			}
			return nil
		})
	})

	// a worker receives an snapshot ID from ch, loads the snapshot
	// and runs fn with id, the snapshot and the error
	worker := func() error {
		for id := range ch {
			debug.Log("load snapshot %v", id)
			sn, err := LoadSnapshot(ctx, loader, id)

			m.Lock()
			err = fn(id, sn, err)
			m.Unlock()
			if err != nil {
				return err
			}
		}
		return nil
	}

	// For most snapshots decoding is nearly for free, thus just assume were only limited by IO
	for i := 0; i < int(loader.Connections()); i++ {
		wg.Go(worker)
	}

	return wg.Wait()
}

func (sn Snapshot) String() string {
	return fmt.Sprintf("",
		sn.id.Str(), sn.Paths, sn.Time, sn.Username, sn.Hostname)
}

// ID returns the snapshot's ID.
func (sn Snapshot) ID() *ID {
	return sn.id
}

func (sn *Snapshot) fillUserInfo() error {
	usr, err := user.Current()
	if err != nil {
		return nil
	}
	sn.Username = usr.Username

	// set userid and groupid
	sn.UID, sn.GID, err = uidGidInt(*usr)
	return err
}

// AddTags adds the given tags to the snapshots tags, preventing duplicates.
// It returns true if any changes were made.
func (sn *Snapshot) AddTags(addTags []string) (changed bool) {
nextTag:
	for _, add := range addTags {
		for _, tag := range sn.Tags {
			if tag == add {
				continue nextTag
			}
		}
		sn.Tags = append(sn.Tags, add)
		changed = true
	}
	return
}

// RemoveTags removes the given tags from the snapshots tags and
// returns true if any changes were made.
func (sn *Snapshot) RemoveTags(removeTags []string) (changed bool) {
	for _, remove := range removeTags {
		for i, tag := range sn.Tags {
			if tag == remove {
				// https://github.com/golang/go/wiki/SliceTricks
				sn.Tags[i] = sn.Tags[len(sn.Tags)-1]
				sn.Tags[len(sn.Tags)-1] = ""
				sn.Tags = sn.Tags[:len(sn.Tags)-1]

				changed = true
				break
			}
		}
	}
	return
}

func (sn *Snapshot) hasTag(tag string) bool {
	for _, snTag := range sn.Tags {
		if tag == snTag {
			return true
		}
	}
	return false
}

// HasTags returns true if the snapshot has all the tags in l.
func (sn *Snapshot) HasTags(l []string) bool {
	for _, tag := range l {
		if tag == "" && len(sn.Tags) == 0 {
			return true
		}
		if !sn.hasTag(tag) {
			return false
		}
	}

	return true
}

// HasTagList returns true if either
// - the snapshot satisfies at least one TagList, so there is a TagList in l
//   for which all tags are included in sn, or
// - l is empty
func (sn *Snapshot) HasTagList(l []TagList) bool {
	debug.Log("testing snapshot with tags %v against list: %v", sn.Tags, l)

	if len(l) == 0 {
		return true
	}

	for _, tags := range l {
		if sn.HasTags(tags) {
			debug.Log("  snapshot satisfies %v %v", tags, l)
			return true
		}
	}

	return false
}

func (sn *Snapshot) hasPath(path string) bool {
	for _, snPath := range sn.Paths {
		if path == snPath {
			return true
		}
	}
	return false
}

// HasPaths returns true if the snapshot has all of the paths.
func (sn *Snapshot) HasPaths(paths []string) bool {
	for _, path := range paths {
		if !sn.hasPath(path) {
			return false
		}
	}

	return true
}

// HasHostname returns true if either
// - the snapshot hostname is in the list of the given hostnames, or
// - the list of given hostnames is empty
func (sn *Snapshot) HasHostname(hostnames []string) bool {
	if len(hostnames) == 0 {
		return true
	}

	for _, hostname := range hostnames {
		if sn.Hostname == hostname {
			return true
		}
	}

	return false
}

// Snapshots is a list of snapshots.
type Snapshots []*Snapshot

// Len returns the number of snapshots in sn.
func (sn Snapshots) Len() int {
	return len(sn)
}

// Less returns true iff the ith snapshot has been made after the jth.
func (sn Snapshots) Less(i, j int) bool {
	return sn[i].Time.After(sn[j].Time)
}

// Swap exchanges the two snapshots.
func (sn Snapshots) Swap(i, j int) {
	sn[i], sn[j] = sn[j], sn[i]
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293