• Apache Kudu 1.15.0部署

    目录

    1. 部署规划

    服务服务器备注
    masterkudu1、kudu2、kudu3
    tablet serverkudu1、kudu2、kudu3

    2. 部署要求

    2.1 硬件要求

    1. 必须是Centos6.4之后的版本才支持Hole Punching，可以通过如下方式进行查看
    [root@kudu1 ~]# uname -r
3.10.0-1160.el7.x86_64
[root@kudu1 ~]#

    • 1
    • 2
    • 3
    1. 必须是ext4或xfs类型的文件系统才支持Hole Punching，可以通过以下方式进行查看
    [root@kudu1 ~]# df -hT
文件系统                       类型      容量  已用  可用 已用% 挂载点
devtmpfs                       devtmpfs  475M     0  475M    0% /dev
tmpfs                          tmpfs     487M     0  487M    0% /dev/shm
tmpfs                          tmpfs     487M  7.7M  479M    2% /run
tmpfs                          tmpfs     487M     0  487M    0% /sys/fs/cgroup
/dev/mapper/centos_centos-root xfs        50G  1.4G   49G    3% /
/dev/mapper/centos_centos-home xfs        47G   33M   47G    1% /home
/dev/sda1                      xfs      1014M  138M  877M   14% /boot
tmpfs                          tmpfs      98M     0   98M    0% /run/user/0
[root@kudu1 ~]# 

    • 1
    • 2
    • 3
    • 4
    • 5
    • 6
    • 7
    • 8
    • 9
    • 10
    • 11

    有时查看的文件系统是ext4格式，但其实是ext2、ext3的设备格式化的，也是不支持Hole Punching的。下面是一个查看磁盘分区属于哪个ext版本的方法，仅供参考

    [root@kudu1 ~]# file -sL /dev/sd*
/dev/sda1: Linux rev 1.0 ext4 filesystem data, UUID=63fa0104-4aab-4dc8-a50d-e2c1bf0fb188 (extents) (large files) (huge files)
/dev/sdb1: Linux rev 1.0 ext2 filesystem data, UUID=b3c82023-78e1-4ad4-b6e0-62355b272166
/dev/sdb2: Linux/i386 swap file (new style), version 1 (4K pages), size 4194303 pages, no label, UUID=3f64308c-19db-4da5-a9a0-db4d7defb80f
[root@kudu1 ~]# 

    • 1
    • 2
    • 3
    • 4
    • 5
    1. Kudu集群的各个服务器之间的时间必须保持同步。参考Centos7服务器通过Chrony设置时间同步

    2.2 软件要求

    每台服务器都安装Java8，参考centos7同时安装java8和openJdk11、windows同时安装java8和openJdk11

    2.3 安装依赖

    [root@kudu1 ~]# yum install gcc cyrus-sasl-plain

    • 1

    3. 从源码编译(kudu1上操作)

    3.1 安装Red Hat Developer Toolset

    为了能够使用C++ 17编译器，需要安装Red Hat Developer Toolset

    [root@kudu1 ~]#
[root@kudu1 ~]# yum install centos-release-scl-rh
[root@kudu1 ~]#
[root@kudu1 ~]# yum install devtoolset-8
[root@kudu1 ~]#

    • 1
    • 2
    • 3
    • 4
    • 5

    在/root/.bashrc添加如下内容，并执行source /root/.bashrc

    source /opt/rh/devtoolset-8/enable

    • 1

    从下列命令结果可以看到，gcc已经是版本8了

    [root@kudu1 ~]# 
[root@kudu1 ~]# gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/opt/rh/devtoolset-8/root/usr/libexec/gcc/x86_64-redhat-linux/8/lto-wrapper
Target: x86_64-redhat-linux
Configured with: ../configure --enable-bootstrap --enable-languages=c,c++,fortran,lto --prefix=/opt/rh/devtoolset-8/root/usr --mandir=/opt/rh/devtoolset-8/root/usr/share/man --infodir=/opt/rh/devtoolset-8/root/usr/share/info --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-shared --enable-threads=posix --enable-checking=release --enable-multilib --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --enable-linker-build-id --with-gcc-major-version-only --with-linker-hash-style=gnu --with-default-libstdcxx-abi=gcc4-compatible --enable-plugin --enable-initfini-array --with-isl=/builddir/build/BUILD/gcc-8.3.1-20190311/obj-x86_64-redhat-linux/isl-install --disable-libmpx --enable-gnu-indirect-function --with-tune=generic --with-arch_32=x86-64 --build=x86_64-redhat-linux
Thread model: posix
gcc version 8.3.1 20190311 (Red Hat 8.3.1-3) (GCC) 
[root@kudu1 ~]#

    • 1
    • 2
    • 3
    • 4
    • 5
    • 6
    • 7
    • 8
    • 9
    • 10

    3.2 安装依赖包

    [root@kudu1 ~]# 
[root@kudu1 ~]# yum install autoconf automake cyrus-sasl-devel cyrus-sasl-gssapi \
> cyrus-sasl-plain flex gcc gcc-c++ gdb git java-1.8.0-openjdk-devel \
> krb5-server krb5-workstation libtool make openssl-devel patch \
> pkgconfig redhat-lsb-core rsync unzip vim-common which
[root@kudu1 ~]# 

    • 1
    • 2
    • 3
    • 4
    • 5
    • 6

    3.3 安装memkind(可选)

    如果需要Kudu的NVM(non-volatile memory)block cache支持，需要安装1.8.0以上的memkind

    [root@kudu1 ~]# yum list memkind
已加载插件：fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirrors.huaweicloud.com
 * centos-sclo-rh: mirrors.cn99.com
 * epel: ftp.iij.ad.jp
 * extras: mirrors.163.com
 * updates: mirrors.163.com
可安装的软件包
memkind.x86_64                                                                       1.7.0-1.el7                                                                       base
[root@kudu1 ~]#

    • 1
    • 2
    • 3
    • 4
    • 5
    • 6
    • 7
    • 8
    • 9
    • 10
    • 11

    linux提供的版本是1.7.0，需要手动编译更高版本的

    [root@kudu1 ~]# yum install numactl-libs numactl-devel
[root@kudu1 ~]#
[root@kudu1 ~]# wget -O memkind-1.13.0.tar.gz https://github.com/memkind/memkind/archive/refs/tags/v1.13.0.tar.gz
[root@kudu1 ~]# 
[root@kudu1 ~]# tar -zxvf memkind-1.13.0.tar.gz
[root@kudu1 ~]# 
[root@kudu1 ~]# cd memkind-1.13.0
[root@kudu1 memkind-1.13.0]# 
[root@kudu1 memkind-1.13.0]# ./autogen.sh
[root@kudu1 memkind-1.13.0]# 
[root@kudu1 memkind-1.13.0]# ./configure --prefix=/usr
[root@kudu1 memkind-1.13.0]#
[root@kudu1 memkind-1.13.0]# make
[root@kudu1 memkind-1.13.0]#
[root@kudu1 memkind-1.13.0]# make install
[root@kudu1 memkind-1.13.0]#
[root@kudu1 memkind-1.13.0]# man memkind
[root@kudu1 memkind-1.13.0]#

    • 1
    • 2
    • 3
    • 4
    • 5
    • 6
    • 7
    • 8
    • 9
    • 10
    • 11
    • 12
    • 13
    • 14
    • 15
    • 16
    • 17
    • 18

    3.4 编译documentation的依赖包(可选)

    暂不安装

    3.5 编译Kudu

    [root@kudu1 ~]#
[root@kudu1 ~]# wget -O kudu-1.15.0.tar.gz https://github.com/apache/kudu/archive/refs/tags/1.15.0.tar.gz
[root@kudu1 ~]#
[root@kudu1 ~]# tar -zxvf kudu-1.15.0.tar.gz
[root@kudu1 ~]#
[root@kudu1 ~]# cd kudu-1.15.0
[root@kudu1 kudu-1.15.0]#

    • 1
    • 2
    • 3
    • 4
    • 5
    • 6
    • 7

    编译第三方依赖包

    [root@kudu1 kudu-1.15.0]# 
[root@kudu1 kudu-1.15.0]# build-support/enable_devtoolset.sh thirdparty/build-if-necessary.sh
......省略部分......
Running post-flight checks
-------------------------
Checking that no TSAN dependency depends on libstdc++ ... PASSED
-------------------------
Post-flight checks succeeded.
+ echo ---------------------
---------------------
+ echo 'Thirdparty dependencies '\''common uninstrumented'\'' built and installed successfully'
Thirdparty dependencies 'common uninstrumented' built and installed successfully
+ exit 0
[root@kudu1 kudu-1.15.0]#

    • 1
    • 2
    • 3
    • 4
    • 5
    • 6
    • 7
    • 8
    • 9
    • 10
    • 11
    • 12
    • 13
    • 14

    编译kudu：创建一个编译目录作为中间输出目录(不能是kudu的根目录)

    [root@kudu1 kudu-1.15.0]# 
[root@kudu1 kudu-1.15.0]# mkdir -p build/release
[root@kudu1 kudu-1.15.0]# 
[root@kudu1 kudu-1.15.0]# cd build/release
[root@kudu1 release]# 
[root@kudu1 release]# ../../build-support/enable_devtoolset.sh \
> ../../thirdparty/installed/common/bin/cmake \
> -DCMAKE_BUILD_TYPE=release ../..
......省略部分......
-- Configuring done
-- Generating done
-- Build files have been written to: /root/kudu-1.15.0/build/release
[root@kudu1 release]#
[root@kudu1 release]# make -j4
[root@kudu1 release]# 

    • 1
    • 2
    • 3
    • 4
    • 5
    • 6
    • 7
    • 8
    • 9
    • 10
    • 11
    • 12
    • 13
    • 14
    • 15

    这里如果编译报错误：已杀死 signal terminated program cc1plus，则需要添加服务器的内存资源。且安装目录下面需要150G的空余磁盘空间

    安装executables可执行文件、libraries和headers(可选)

    [root@kudu1 ~]# 
[root@kudu1 ~]# mkdir kudu-1.15.0-install
[root@kudu1 ~]#
[root@kudu1 ~]# cd kudu-1.15.0/build/release/
[root@kudu1 release]#
[root@kudu1 release]# make DESTDIR=/root/kudu-1.15.0-install install
[root@kudu1 release]#

    • 1
    • 2
    • 3
    • 4
    • 5
    • 6
    • 7

    如果不指定DESTDIR，则默认目录为/usr/local，会在DESTDIR目录下生成以下

    • usr/local/sbin目录：kudu-tserver和kudu-master可执行文件
    • usr/local/bin目录：kudu命令行工具
    • usr/local/lib64目录：kudu的client library
    • usr/local/include/kudu目录：kudu的client headers

    编译documentation(可选)

    暂不安装

    4. 配置Kudu(kudu1上操作)

    4.1 配置基础

    kudu-master和kudu-tserver有2种方式传递配置参数

    1. 直接传递配置参数，比如kudu-master --master_addresses=localhost --log_dir=/tmp
    2. 传递一个或多个配置文件，比如kudu-master --flagfile=/tmp/master.gflagfile。配置文件中可以包含配置参数和配置文件
    --master_addresses=localhost
--flagfile=/tmp/tmp.gflagfile

    • 1
    • 2

    查看帮助信息，通过–help获取，比如kudu-master --help

    4.2 拷贝kudu的webserver的CSS文件

    [root@kudu1 local]# 
[root@kudu1 local]# pwd
/root/kudu-1.15.0-install/usr/local
[root@kudu1 local]# 
[root@kudu1 local]# scp -r /root/kudu-1.15.0/www .
[root@kudu1 local]# 

    • 1
    • 2
    • 3
    • 4
    • 5
    • 6

    4.3 创建数据目录

    [root@kudu1 local]#
[root@kudu1 local]# pwd
/root/kudu-1.15.0-install/usr/local
[root@kudu1 local]# 
[root@kudu1 local]# mkdir conf
[root@kudu1 local]# 
[root@kudu1 local]# mkdir -p data/master/data
[root@kudu1 local]# mkdir -p data/master/metadata
[root@kudu1 local]# mkdir -p data/master/wal
[root@kudu1 local]# mkdir -p data/master/log
[root@kudu1 local]# 
[root@kudu1 local]# mkdir -p data/tserver/data
[root@kudu1 local]# mkdir -p data/tserver/metadata
[root@kudu1 local]# mkdir -p data/tserver/wal
[root@kudu1 local]# mkdir -p data/tserver/log
[root@kudu1 local]# 

    • 1
    • 2
    • 3
    • 4
    • 5
    • 6
    • 7
    • 8
    • 9
    • 10
    • 11
    • 12
    • 13
    • 14
    • 15
    • 16

    4.4 conf/master.gflagfile

    新增master.gflagfile，文件内容如下：

    # 所有的Master RPC地址，默认localhost
--master_addresses=kudu1:7051,kudu2:7051,kudu3:7051

# master储存data blocks的目录，多个逗号分隔
--fs_data_dirs=/root/kudu-1.15.0-install/usr/local/data/master/data

# master储存tablet metadata的目录
--fs_metadata_dir=/root/kudu-1.15.0-install/usr/local/data/master/metadata

# master储存write-ahead logs的目录
--fs_wal_dir=/root/kudu-1.15.0-install/usr/local/data/master/wal

# master储存log files的目录，默认/tmp
--log_dir=/root/kudu-1.15.0-install/usr/local/data/master/log

# 指定master的webserver的CSS文件
--webserver_doc_root=/root/kudu-1.15.0-install/usr/local/www

# 指定master的block cache大小，默认512
--block_cache_capacity_mb=128

# 允许任何服务器的IP地址进行访问master，默认为127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,169.254.0.0/16
--trusted_subnets=0.0.0.0/0


    • 1
    • 2
    • 3
    • 4
    • 5
    • 6
    • 7
    • 8
    • 9
    • 10
    • 11
    • 12
    • 13
    • 14
    • 15
    • 16
    • 17
    • 18
    • 19
    • 20
    • 21
    • 22
    • 23
    • 24

    4.5 conf/tserver.gflagfile

    新增tserver.gflagfile，文件内容如下：

    # 连接的Master RPC地址，默认127.0.0.1:7051
--tserver_master_addrs=kudu1:7051,kudu2:7051,kudu3:7051

# tserver储存data blocks的目录，多个逗号分隔
--fs_data_dirs=/root/kudu-1.15.0-install/usr/local/data/tserver/data

# tserver储存tablet metadata的目录
--fs_metadata_dir=/root/kudu-1.15.0-install/usr/local/data/tserver/metadata

# tserver储存write-ahead logs的目录
--fs_wal_dir=/root/kudu-1.15.0-install/usr/local/data/tserver/wal

# tserver储存log files的目录，默认/tmp
--log_dir=/root/kudu-1.15.0-install/usr/local/data/tserver/log

# 分配给tablet server的block cache最大内存，默认512
--block_cache_capacity_mb=256

# tablet server能使用的最大内存，超过该值就拒绝write请求，默认4294967296
--memory_limit_hard_bytes=1073741824

# 指定master的webserver的CSS文件
--webserver_doc_root=/root/kudu-1.15.0-install/usr/local/www

# 允许任何服务器的IP地址进行访问tserver，默认为127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,169.254.0.0/16
--trusted_subnets=0.0.0.0/0


    • 1
    • 2
    • 3
    • 4
    • 5
    • 6
    • 7
    • 8
    • 9
    • 10
    • 11
    • 12
    • 13
    • 14
    • 15
    • 16
    • 17
    • 18
    • 19
    • 20
    • 21
    • 22
    • 23
    • 24
    • 25
    • 26
    • 27

    4.6 Tables配置

    可以按表粒度设置，设置方式有2种

    1. 创建表的时候设置
    2. 通过API或命令行进行修改
    Configuration类型默认值描述
    kudu.table.history_max_age_secintegerNumber of seconds to retain history for tablets in this table
    kudu.table.maintenance_priorityinteger0Priority level of a table for maintenance

    5. 分发kudu安装文件(kudu1上操作)

    [root@kudu1 ~]# scp -r kudu-1.15.0-install root@kudu2:/root
[root@kudu1 ~]# scp -r kudu-1.15.0-install root@kudu3:/root

    • 1
    • 2

    6. 启动(3台服务器)

    6.1 启动master

    因为启动一个master服务，需要连接其它的master服务，所以依次迅速的启动各个master

    [root@kudu1 ~]# nohup kudu-1.15.0-install/usr/local/sbin/kudu-master --flagfile=kudu-1.15.0-install/usr/local/conf/master.gflagfile > kudu-1.15.0-install/usr/local/sbin/kudu-master.log 2>&1 &
[1] 9251
[root@kudu1 ~]#

    • 1
    • 2
    • 3

    分别访问http://kudu1:8051、http://kudu2:8051、http://kudu3:8051。其中http://kudu1:8051的页面如下所示：
    kudu master

    6.2 启动tserver

    [root@kudu1 ~]# nohup kudu-1.15.0-install/usr/local/sbin/kudu-tserver --flagfile=kudu-1.15.0-install/usr/local/conf/tserver.gflagfile > kudu-1.15.0-install/usr/local/sbin/kudu-tserver.log 2>&1 &
[2] 20995
[root@kudu1 ~]#

    • 1
    • 2
    • 3

    分别访问http://kudu1:8050、http://kudu2:8050、http://kudu3:8050。其中http://kudu1:8050的页面如下所示：
    kudu tserver

    6.3 查看集群状态和数据完整性

    ksck会发现问题。例如无法连接的tablet server、没有leader的tablet 、副本不足的tablet

    查看集群状态命令

    [root@kudu1 local]# pwd
/root/kudu-1.15.0-install/usr/local
[root@kudu1 local]# 
[root@kudu1 local]# bin/kudu cluster ksck kudu1:7051,kudu2:7051,kudu3:7051
Master Summary
               UUID               |     Address     | Status
----------------------------------+-----------------+---------
 62275082a3ab4be7a474c8ade4429616 |   kudu1:7051    | HEALTHY
 9ba2cdb6fc65440196af83d8ea3d52b8 |   kudu2:7051    | HEALTHY
 9ba2cdb6fc65440196af83d8ea3d52b9 |   kudu3:7051    | HEALTHY

Flags of checked categories for Master:
        Flag         |                            Value                            |         Master
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
 time_source         | system                                                      | all 3 server(s) checked

Tablet Server Summary
               UUID               |     Address     | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+-----------------+---------+----------+----------------+-----------------
 7c6312b0200141d1ab7bde4990b2b577 |   kudu1:7050    | HEALTHY | <none>   |       0        |       0
 f21688849b7e4e1a901b2e580aba8e23 |   kudu2:7050    | HEALTHY | <none>   |       0        |       0
 f21688849b7e4e1a901b2e580aba8e24 |   kudu3:7050    | HEALTHY | <none>   |       0        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       3

Flags of checked categories for Tablet Server:
        Flag         |                            Value                            |      Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
 time_source         | system                                                      | all 3 server(s) checked

Version Summary
 Version |         Servers
---------+-------------------------
 1.15.0  | all 6 server(s) checked

The cluster doesn't have any matching tablets

The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 0
 First Quartile | 0
 Median         | 0
 Third Quartile | 0
 Maximum        | 0


OK
[root@kudu1 local]# 

    • 1
    • 2
    • 3
    • 4
    • 5
    • 6
    • 7
    • 8
    • 9
    • 10
    • 11
    • 12
    • 13
    • 14
    • 15
    • 16
    • 17
    • 18
    • 19
    • 20
    • 21
    • 22
    • 23
    • 24
    • 25
    • 26
    • 27
    • 28
    • 29
    • 30
    • 31
    • 32
    • 33
    • 34
    • 35
    • 36
    • 37
    • 38
    • 39
    • 40
    • 41
    • 42
    • 43
    • 44
    • 45
    • 46
    • 47
    • 48
    • 49
    • 50
    • 51
    • 52
    • 53
    • 54
    • 55
    • 56

    查看数据完整性命令
    将一个tablet的所有副本数据进行计较，检验数据完整性

    [root@kudu1 local]# 
[root@kudu1 local]# bin/kudu cluster ksck --checksum_scan --tables kudu_java_test --tablets 3fe1744a20e7433d864bb808a655c277,447b26717a33478abb4f004651e34aaf kudu1:7051,kudu2:7051,kudu3:7051
Checksum finished in 0s: 0/3 replicas remaining (0B from disk, 0 rows summed)
Master Summary
               UUID               |     Address     | Status
----------------------------------+-----------------+---------
 62275082a3ab4be7a474c8ade4429616 |    kudu1:7051   | HEALTHY
 9ba2cdb6fc65440196af83d8ea3d52b8 |    kudu2:7051   | HEALTHY
 9ba2cdb6fc65440196af83d8ea3d52b8 |    kudu3:7051   | HEALTHY

Flags of checked categories for Master:
        Flag         |                            Value                            |         Master
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
 time_source         | system                                                      | all 3 server(s) checked

Tablet Server Summary
               UUID               |     Address     | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+-----------------+---------+----------+----------------+-----------------
 7c6312b0200141d1ab7bde4990b2b577 |    kudu1:7050   | HEALTHY | <none>   |       6        |       0
 f21688849b7e4e1a901b2e580aba8e23 |    kudu2:7050   | HEALTHY | <none>   |       6        |       0
 f21688849b7e4e1a901b2e580aba8e23 |    kudu3:7050   | HEALTHY | <none>   |       6        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       3

Flags of checked categories for Tablet Server:
        Flag         |                            Value                            |      Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
 time_source         | system                                                      | all 3 server(s) checked

Version Summary
 Version |         Servers
---------+-------------------------
 1.15.0  | all 6 server(s) checked

Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
      Name      | RF | Status  | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------+----+---------+---------------+---------+------------+------------------+-------------
 kudu_java_test | 1  | HEALTHY | 3             | 3       | 0          | 0                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 0
 First Quartile | 0
 Median         | 3
 Third Quartile | 3
 Maximum        | 3

Checksum Summary
-----------------------
kudu_java_test
-----------------------
T 3fe1744a20e7433d864bb808a655c277 P f21688849b7e4e1a901b2e580aba8e23 (kudu3:7050): Checksum: 0
T 447b26717a33478abb4f004651e34aaf P f21688849b7e4e1a901b2e580aba8e23 (kudu3:7050): Checksum: 0

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 3
 Tablet Servers | 3
 Tables         | 1
 Tablets        | 3
 Replicas       | 3

OK
[root@kudu1 local]# 

    • 1
    • 2
    • 3
    • 4
    • 5
    • 6
    • 7
    • 8
    • 9
    • 10
    • 11
    • 12
    • 13
    • 14
    • 15
    • 16
    • 17
    • 18
    • 19
    • 20
    • 21
    • 22
    • 23
    • 24
    • 25
    • 26
    • 27
    • 28
    • 29
    • 30
    • 31
    • 32
    • 33
    • 34
    • 35
    • 36
    • 37
    • 38
    • 39
    • 40
    • 41
    • 42
    • 43
    • 44
    • 45
    • 46
    • 47
    • 48
    • 49
    • 50
    • 51
    • 52
    • 53
    • 54
    • 55
    • 56
    • 57
    • 58
    • 59
    • 60
    • 61
    • 62
    • 63
    • 64
    • 65
    • 66
    • 67
    • 68
    • 69
    • 70
    • 71
    • 72
    • 73
  • 相关阅读:
    华为配置CAPWAP双栈覆盖业务示例
    隐私计算python实现Paillier同态加密
    wifi7有关的210个提案
    基于SpringBoot的家电销售电商管理平台
    promise时效架构升级方案的实施及落地 | 京东物流技术团队
    【Python】高级变量类型
    总结linux常用命令
    【JavaEE--复习】二、整合数据层MyBatis
    链表|707.设计链表
    感受野计算问题
  • 原文地址：https://blog.csdn.net/yy8623977/article/details/123518208