配置iSCSI实现Linux的远程块存储

一 名词解释

二 需求

1）在服务端创建一个1GB大小的分区用作后备存储。
2）创建名为iqn.2022-11.com.example:serverx的新 ISCSI 目标，并且仅限于iqn.2022-11.com.example:desktopx的客户端。
3）发现并登录到新目标。
4）在新磁盘上创建 XFS 文件系统，并将其持久挂载到 /iscsidisk上。

三 环境准备

服务端IP10.0.0.11 主机名：server
客户端IP10.0.0.12 主机名：client
使用红帽旗下的 centos7.6 系统

[root@server tools]# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.6 (Maipo)
[root@server tools]# uname -r
3.10.0-957.el7.x86_64
1
2
3
4

关闭了防火墙和 selinux

[root@server tools]# systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
   Active: inactive (dead)
     Docs: man:firewalld(1)
[root@server tools]# getenforce
Disabled
1
2
3
4
5
6
7

四 搭建流程

1 服务端准备硬盘并分区

Linux磁盘分区和管理

准备一块10G的硬盘，添加硬盘并让系统检测到，再进行一次分区，分区为1个G。
如果不清楚如何磁盘分区，可以参考笔者的写过的文章（上面是链接），此处不作重点阐述，只进行命令的展示。

[root@server ~]# fdisk /dev/sdb
Welcome to fdisk (util-linux 2.23.2).

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0x6e73b1a0.

Command (m for help): n
Partition type:
   p   primary (0 primary, 0 extended, 4 free)
   e   extended
Select (default p): p
Partition number (1-4, default 1): 1
First sector (2048-20971519, default 2048): 
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-20971519, default 20971519): +1G
Partition 1 of type Linux and of size 1 GiB is set

Command (m for help): p

Disk /dev/sdb: 10.7 GB, 10737418240 bytes, 20971520 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x6e73b1a0

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1            2048     2099199     1048576   83  Linux

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
[root@server ~]# partprobe /dev/sdb
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

2 服务端安装软件包，并启动服务

[root@server ~]# yum install -y targetcli
Loaded plugins: langpacks, product-id, search-disabled-repos, subscription-manager
This system is not registered with an entitlement server. You can use subscription-manager to register.
epel

================================                                                                                            
中间信息省略
================================

Installed:
  targetcli.noarch 0:2.1.53-1.el7_9                                                                                             

Dependency Installed:
  pyparsing.noarch 0:1.5.6-9.el7             python-configshell.noarch 1:1.1.26-1.el7      python-kmod.x86_64 0:0.9-4.el7     
  python-rtslib.noarch 0:2.1.74-1.el7_9      python-urwid.x86_64 0:1.1.1-3.el7            

Complete!
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

安装后，启动服务，开机后自动启动服务，并检查服务状态

[root@server ~]# systemctl start target
[root@server ~]# systemctl enable target
Created symlink from /etc/systemd/system/multi-user.target.wants/target.service to /usr/lib/systemd/system/target.service.
[root@server ~]# systemctl status target
● target.service - Restore LIO kernel target configuration
   Loaded: loaded (/usr/lib/systemd/system/target.service; enabled; vendor preset: disabled)
   Active: active (exited) since Sun 2022-11-27 20:15:37 CST; 9s ago
 Main PID: 13609 (code=exited, status=0/SUCCESS)

Nov 27 20:15:36 server systemd[1]: Starting Restore LIO kernel target configuration...
Nov 27 20:15:37 server target[13609]: No saved config file at /etc/target/saveconfig.json, ok, exiting
Nov 27 20:15:37 server systemd[1]: Started Restore LIO kernel target configuration.
1
2
3
4
5
6
7
8
9
10
11
12

3 服务端 5个create 操作

首先进入targetcli交互式模式

[root@server ~]# targetcli
Warning: Could not load preferences file /root/.targetcli/prefs.bin.
targetcli shell version 2.1.53
Copyright 2011-2013 by Datera, Inc and others.
For help on commands, type 'help'.

/> 
1
2
3
4
5
6
7

3-1 定义块设备，创建后备存储。

block1 是创建的后端存储名称， /dev/sdb1是块存储设备

/> cd /backstores/block 
/backstores/block> create block1 /dev/sdb1
Created block storage object block1 using /dev/sdb1.
1
2
3

3-2 为目标创建IQN

/backstores/block> cd /iscsi 
/iscsi> create iqn.2022-11.com.example:serverx
Created target iqn.2022-11.com.example:serverx.
Created TPG 1.
Global pref auto_add_default_portal=true
Created default portal listening on all IPs (0.0.0.0), port 3260.
1
2
3
4
5
6

3-3 创建一个ACL来控制特定的客户端访问

此处的IQN是客户端/etc/iscsi/initiatorname.iscsi文件里面的内容，表示本地服务端允许哪些客户端可以访问的白名单。

/iscsi> cd iqn.2022-11.com.example:serverx/tpg1/acls 
/iscsi/iqn.20...erx/tpg1/acls> create iqn.2022-11.com.example:desktopx
Created Node ACL for iqn.2022-11.com.example:desktopx
1
2
3

3-4 为每个后备存储创建一个LUN，激活后备存储，由ACL自动分配

/iscsi/iqn.20...erx/tpg1/acls> cd /iscsi/iqn.2022-11.com.example:serverx/tpg1/luns 
/iscsi/iqn.20...erx/tpg1/luns> create /backstores/block/block1 
Created LUN 0.
Created LUN 0->0 mapping in node ACL iqn.2022-11.com.example:desktopx
1
2
3
4

3-5 创建门户来指定（服务端）本地要侦听的IP和端口

此处指的是服务端开放给客户端可以找到的IP和端口，因此写的是服务端自己的IP和选择开放的端口，注意不要和本地的端口冲突。本机（服务端）的IP是10.0.0.11，因此开放的IP和端口是10.0.0.11 3260

/iscsi/iqn.20...erx/tpg1/luns> cd /iscsi/iqn.2022-11.com.example:serverx/tpg1/portals
/iscsi/iqn.20.../tpg1/portals> create 10.0.0.11 3260
Using default IP port 3260
Could not create NetworkPortal in configFS
/iscsi/iqn.20.../tpg1/portals> delete 0.0.0.0 3260
Deleted network portal 0.0.0.0:3260
/iscsi/iqn.20.../tpg1/portals> create 10.0.0.11 3260
Using default IP port 3260
Created network portal 10.0.0.11:3260.
1
2
3
4
5
6
7
8
9

可以看到，由于本地的端口冲突，我删除了0.0.0.0 3260的IP和端口，添加了自己设置的内容。

3-6 保存配置信息并退出

使用ctrl+D即可保存退出。

/> exit
Global pref auto_save_on_exit=true
Configuration saved to /etc/target/saveconfig.json
1
2
3

3-7 查看服务端整体配置信息

4 客户端安装软件，修改IQN信息，启动服务

安装软件包

[root@client iscsi]# yum install -y iscsi-initiator-utils
Loaded plugins: langpacks, product-id, search-disabled-repos, subscription-manager
This system is not registered with an entitlement server. You can use subscription-manager to register.
epel                                                                                                     

==================
中间信息省略
==================

Updated:
  iscsi-initiator-utils.x86_64 0:6.2.0.874-22.el7_9                                                                             

Dependency Updated:
  iscsi-initiator-utils-iscsiuio.x86_64 0:6.2.0.874-22.el7_9                                                                    

Complete!
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

修改IQN信息。

（如果客户端ISCSI服务原本是开启的，修改完配置文件后，需要重启 iscsid服务。）

[root@client tools]# cd /etc/iscsi/
[root@client iscsi]# cp initiatorname.iscsi initiatorname.iscsi.bak
[root@client iscsi]# cat initiatorname.iscsi
InitiatorName=iqn.2022-11.com.example:desktopx
1
2
3
4

启动服务，实现服务的开机自动启动，检查服务状态

[root@client iscsi]# systemctl start iscsid
[root@client iscsi]# systemctl enable iscsid
[root@client iscsi]# systemctl status iscsid
● iscsid.service - Open-iSCSI
   Loaded: loaded (/usr/lib/systemd/system/iscsid.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2022-11-27 21:01:33 CST; 8min ago
     Docs: man:iscsid(8)
           man:iscsiuio(8)
           man:iscsiadm(8)
 Main PID: 101685 (iscsid)
   Status: "Ready to process requests"
   CGroup: /system.slice/iscsid.service
           └─101685 /sbin/iscsid -f

Nov 27 21:01:33 client systemd[1]: Starting Open-iSCSI...
Nov 27 21:01:33 client systemd[1]: Started Open-iSCSI.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

5 发现目标和登录目标

发现目标

[root@client iscsi]# iscsiadm -m discovery -t sendtargets -p 10.0.0.11
10.0.0.11:3260,1 iqn.2022-11.com.example:serverx
1
2

登陆目标

[root@client iscsi]# iscsiadm -m node -T iqn.2022-11.com.example:serverx -p 10.0.0.11 -l
Logging in to [iface: default, target: iqn.2022-11.com.example:serverx, portal: 10.0.0.11,3260] (multiple)
Login to [iface: default, target: iqn.2022-11.com.example:serverx, portal: 10.0.0.11,3260] successful.
1
2
3

ps.如果想要删除这些目标的发现，则执行下面命令，先登陆退出，再删除。
（当你完成本文的需求后，想要删除的话，可以执行下面两个命令操作）。

[root@client ~]# iscsiadm -m node -T iqn.2022-11.com.example:serverx -p 10.0.0.11 --logout
Logging out of session [sid: 1, target: iqn.2022-11.com.example:serverx, portal: 10.0.0.11,3260]
Logout of [sid: 1, target: iqn.2022-11.com.example:serverx, portal: 10.0.0.11,3260] successful.
[root@client ~]# iscsiadm -m node -o delete
1
2
3
4

6 客户端挂载文件系统

查看当前磁盘信息，会发现多了一块盘

[root@client iscsi]# lsblk
NAME          MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda             8:0    0   20G  0 disk 
├─sda1          8:1    0  200M  0 part /boot
└─sda2          8:2    0 19.8G  0 part 
  ├─rhel-root 253:0    0 18.3G  0 lvm  /
  └─rhel-swap 253:1    0  1.5G  0 lvm  [SWAP]
sdb             8:16   0    1G  0 disk 
sr0            11:0    1  4.2G  0 rom  
[root@client iscsi]# ll /dev/sdb
brw-rw---- 1 root disk 8, 16 Nov 27 21:12 /dev/sdb
1
2
3
4
5
6
7
8
9
10
11

首先创建文件系统

[root@client iscsi]# mkfs -t xfs /dev/sdb
meta-data=/dev/sdb               isize=512    agcount=4, agsize=65536 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=0, sparse=0
data     =                       bsize=4096   blocks=262144, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal log           bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
1
2
3
4
5
6
7
8
9
10

检查文件系统创建的信息

[root@client iscsi]# lsblk /dev/sdb
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sdb    8:16   0   1G  0 disk 
1
2
3

创建挂载点，并进行设备的临时挂载

[root@client iscsi]# mkdir /iscsidisk
[root@client iscsi]# mount /dev/sdb /iscsidisk
1
2

将挂载信息写入配置文件，实现永久挂载

[root@client iscsi]# tail -1 /etc/fstab 
/dev/sdb                /iscsidisk              xfs     defaults        0 0
1
2

最后将所有目录重新挂载，再次确认

[root@client iscsi]# mount -a
[root@client iscsi]# df -h
Filesystem             Size  Used Avail Use% Mounted on
/dev/mapper/rhel-root   19G  4.1G   15G  23% /
devtmpfs               470M     0  470M   0% /dev
tmpfs                  487M     0  487M   0% /dev/shm
tmpfs                  487M  8.1M  479M   2% /run
tmpfs                  487M     0  487M   0% /sys/fs/cgroup
/dev/sda1              197M  143M   55M  73% /boot
tmpfs                   98M     0   98M   0% /run/user/0
/dev/sdb              1014M   33M  982M   4% /iscsidisk
1
2
3
4
5
6
7
8
9
10
11