• MongoShake迁移mongodb数据库

    MongoShake简介

    MongoShake是阿里云以Golang语言编写的通用平台型服务工具，完全开源，它通过读取MongoDB的Oplog操作日志来复制MongoDB的数据以实现特定需求。

    基于mongodb oplog的集群复制工具，可以满足迁移和同步的需求，进一步实现灾备和多活功能。可以实现MongoDB数据库间的数据同步，该功能可用于数据分析、灾备和多活等业务场景。

    在这里插入图片描述

    MongoShake还提供了日志数据的订阅和消费功能，可通过SDK、Kafka、MetaQ等方式的灵活对接，适用于日志订阅、数据中心同步、Cache异步淘汰等场景。

    项目地址：https://github.com/alibaba/MongoShake
    参考文档：https://help.aliyun.com/document_detail/122621.html

    mongodb 数据迁移

    1、在kubernetes集群中创建2个示例 mongodb 副本集集群，作为迁移源端和目标端。

    添加bitnami mongodb helm仓库

    helm repo add bitnami https://charts.bitnami.com/bitnami

    • 1

    部署第一个集群

    helm upgrade -i mongodb bitnami/mongodb -n mongodb --create-namespace \
  --set architecture=replicaset \
  --set replicaCount=2 \
  --set externalAccess.enabled=true \
  --set externalAccess.service.type=NodePort \
  --set externalAccess.service.nodePorts[0]='30700' \
  --set externalAccess.service.nodePorts[1]='30701'

    • 1
    • 2
    • 3
    • 4
    • 5
    • 6
    • 7

    部署第二个集群

    helm upgrade -i mongodb-dest bitnami/mongodb -n mongodb-dest --create-namespace \
  --set architecture=replicaset \
  --set replicaCount=2 \
  --set externalAccess.enabled=true \
  --set externalAccess.service.type=NodePort \
  --set externalAccess.service.nodePorts[0]='30702' \
  --set externalAccess.service.nodePorts[1]='30703'

    • 1
    • 2
    • 3
    • 4
    • 5
    • 6
    • 7

    获取第一个集群的root密码

    export MONGODB_ROOT_PASSWORD=$(kubectl get secret --namespace mongodb-dest mongodb-dest -o jsonpath="{.data.mongodb-root-password}" | base64 -d)
echo $MONGODB_ROOT_PASSWORD

    • 1
    • 2

    安装mongodb客户端工具，连接到第一个mongodb集群，连接方式为nodeip+nodeport，地址为pod所在节点地址。

    [root@Ops-repo mongo-shake]# mongo --host "192.168.0.23:30700,192.168.0.24:30701" --authenticationDatabase admin -u root -p K1IySNf3JM
MongoDB shell version v4.4.4
connecting to: mongodb://192.168.0.23:30700,192.168.0.24:30701/admin?authSource=admin&compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("71963472-8169-4248-8253-fbde1d981d39") }
MongoDB server version: 6.0.2
WARNING: shell and server versions do not match
---
The server generated these startup warnings when booting: 
        2022-11-01T05:14:02.656+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem
        2022-11-01T05:14:05.060+00:00: vm.max_map_count is too low
        2022-11-01T05:14:05.060+00:00:         currentValue: 262144
        2022-11-01T05:14:05.060+00:00:         recommendedMinimum: 1677720
        2022-11-01T05:14:05.060+00:00:         maxConns: 838860
---
---
        Enable MongoDB's free cloud-based monitoring service, which will then receive and display
        metrics about your deployment (disk utilization, CPU, operation statistics, etc).

        The monitoring data will be available on a MongoDB website with a unique URL accessible to you
        and anyone you share the URL with. MongoDB may use this information to make product
        improvements and to suggest MongoDB products and deployment options to you.

        To enable free monitoring, run the following command: db.enableFreeMonitoring()
        To permanently disable this reminder, run the following command: db.disableFreeMonitoring()
---
rs0:PRIMARY> 

    • 1
    • 2
    • 3
    • 4
    • 5
    • 6
    • 7
    • 8
    • 9
    • 10
    • 11
    • 12
    • 13
    • 14
    • 15
    • 16
    • 17
    • 18
    • 19
    • 20
    • 21
    • 22
    • 23
    • 24
    • 25
    • 26

    迁移源端创建示例数据

    > use testdb;
> db.createCollection("users");
> db.myusers.insert({"name":"jack", age:24});

    • 1
    • 2
    • 3

    MongoShake安装

    1、安装MongoShake程序

    wget https://github.com/alibaba/MongoShake/releases/download/release-v2.8.1-20220922/mongo-shake-v2.8.1.tgz

    • 1

    执行如下命令将MongoShake解压到/opt/mongoshake目录中。

    tar zxvf mongo-shake-v2.8.1.tgz -C /opt
mv /opt/mongo-shake-v2.8.1/ /opt/mongo-shake
cd /opt/mongo-shake

    • 1
    • 2
    • 3

    2、修改MongoShake的配置文件collector.conf，涉及的主要参数说明如下所示。

    说明：密码中不得包含艾特（@）字符，否则会导致连接失败。

    mongo_urls 为源端MongoDB实例的ConnectionStringURI格式连接地址。示例如下：

    mongo_urls = mongodb://root:K1IySNf3JM@192.168.0.23:30700,192.168.0.24:30701

    • 1

    tunnel.address为目标端MongoDB实例的ConnectionStringURI格式连接地址。示例如下：

    tunnel.address = mongodb://root:o72ntfVFVL@192.168.0.23:30702,192.168.0.24:30703

    • 1

    sync_mode 为数据同步的方式，默认值为incr。all表示全量+增量同步，full表示全量同步，incr表示增量同步。

    sync_mode = incr

    • 1

    执行迁移，执行下述命令启动同步任务，并打印日志信息。

    ./collector.linux -conf=collector.conf -verbose

    • 1

    观察打印的日志信息，当出现如下日志时，即代表全量数据同步已完成，并进入增量数据同步模式。

    [root@Ops-repo mongo-shake]# ./collector.linux -conf=collector.conf -verbose 2
[2022/11/01 13:45:36 CST] [INFO] log init succ. log.dir[] log.name[collector.log] log.level[info]
[2022/11/01 13:45:36 CST] [INFO] MongoDB Version Source[6.0.2] Target[6.0.2]
[2022/11/01 13:45:36 CST] [WARN] 
______________________________
\                             \           _         ______ |
 \                             \        /   \___-=O'/|O'/__|
  \  MongoShake, Here we go !!  \_______\          / | /    )
  /                             /        '/-==__ _/__|/__=-|  -GM
 /        Alibaba Cloud        /         *             \ | |
/                             /                        (o)
------------------------------

if you have any problem, please visit https://github.com/alibaba/MongoShake/wiki/FAQ

[2022/11/01 13:45:36 CST] [INFO] New session to mongodb://root:***@192.168.0.23:30700,192.168.0.24:30701 successfully
[2022/11/01 13:45:36 CST] [INFO] Close client with mongodb://root:***@192.168.0.23:30700,192.168.0.24:30701
[2022/11/01 13:45:36 CST] [INFO] New session to mongodb://root:***@192.168.0.23:30700,192.168.0.24:30701 successfully
[2022/11/01 13:45:36 CST] [INFO] Close client with mongodb://root:***@192.168.0.23:30700,192.168.0.24:30701
[2022/11/01 13:45:36 CST] [INFO] Collector startup. shard_by[collection] gids[[]]
[2022/11/01 13:45:36 CST] [INFO] Collector configuration {"ConfVersion":10,"Id":"mongoshake","MasterQuorum":false,"FullSyncHTTPListenPort":9101,"IncrSyncHTTPListenPort":9100,"SystemProfilePort":9200,"LogLevel":"info","LogDirectory":"","LogFileName":"collector.log","LogFlush":false,"SyncMode":"incr","MongoUrls":["mongodb://root:***@192.168.0.23:30700,192.168.0.24:30701"],"MongoCsUrl":"","MongoSUrl":"","MongoSslRootCaFile":"","MongoSslClientCaFile":"","MongoConnectMode":"secondaryPreferred","Tunnel":"direct","TunnelAddress":["mongodb://root:***@192.168.0.23:30702,192.168.0.24:30703"],"TunnelMessage":"raw","TunnelKafkaPartitionNumber":1,"TunnelJsonFormat":"","TunnelMongoSslRootCaFile":"","FilterNamespaceBlack":[],"FilterNamespaceWhite":[],"FilterPassSpecialDb":[],"FilterDDLEnable":false,"FilterOplogGids":false,"CheckpointStorageUrl":"mongodb://root:***@192.168.0.23:30700,192.168.0.24:30701","CheckpointStorageDb":"mongoshake","CheckpointStorageCollection":"ckpt_default","CheckpointStorageUrlMongoSslRootCaFile":"","CheckpointStartPosition":1,"TransformNamespace":[],"SpecialSourceDBFlag":"","FullSyncReaderCollectionParallel":6,"FullSyncReaderWriteDocumentParallel":8,"FullSyncReaderDocumentBatchSize":128,"FullSyncReaderFetchBatchSize":1024,"FullSyncReaderParallelThread":1,"FullSyncReaderParallelIndex":"_id","FullSyncCollectionDrop":true,"FullSyncCreateIndex":"none","FullSyncReaderOplogStoreDisk":false,"FullSyncReaderOplogStoreDiskMaxSize":256000,"FullSyncExecutorInsertOnDupUpdate":false,"FullSyncExecutorFilterOrphanDocument":false,"FullSyncExecutorMajorityEnable":false,"IncrSyncMongoFetchMethod":"oplog","IncrSyncChangeStreamWatchFullDocument":false,"IncrSyncReaderFetchBatchSize":1024,"IncrSyncOplogGIDS":[],"IncrSyncShardKey":"collection","IncrSyncShardByObjectIdWhiteList":[],"IncrSyncWorker":8,"IncrSyncTunnelWriteThread":8,"IncrSyncTargetDelay":0,"IncrSyncWorkerBatchQueueSize":64,"IncrSyncAdaptiveBatchingMaxSize":1024,"IncrSyncFetcherBufferCapacity":256,"IncrSyncExecutorUpsert":false,"IncrSyncExecutorInsertOnDupUpdate":false,"IncrSyncConflictWriteTo":"none","IncrSyncExecutorMajorityEnable":false,"CheckpointStorage":"database","CheckpointInterval":5000,"FullSyncExecutorDebug":false,"IncrSyncDBRef":false,"IncrSyncExecutor":1,"IncrSyncExecutorDebug":false,"IncrSyncReaderDebug":"","IncrSyncCollisionEnable":false,"IncrSyncReaderBufferTime":1,"IncrSyncWorkerOplogCompressor":"none","IncrSyncTunnelKafkaDebug":"","Version":"improve-2.8.1,9f2515bbe508b2d86fab2ed87f01f3121efc6920,release,go1.15.10,2022-09-22_11:49:22","SourceDBVersion":"6.0.2","TargetDBVersion":"6.0.2","IncrSyncTunnel":"","IncrSyncTunnelAddress":null,"IncrSyncTunnelMessage":"","HTTPListenPort":0,"SystemProfile":0}
[2022/11/01 13:45:36 CST] [INFO] New session to mongodb://root:***@192.168.0.23:30700,192.168.0.24:30701 successfully
[2022/11/01 13:45:36 CST] [INFO] Close client with mongodb://root:***@192.168.0.23:30700,192.168.0.24:30701
[2022/11/01 13:45:36 CST] [INFO] New session to mongodb://root:***@192.168.0.23:30700,192.168.0.24:30701 successfully
[2022/11/01 13:45:36 CST] [INFO] Close client with mongodb://root:***@192.168.0.23:30700,192.168.0.24:30701
[2022/11/01 13:45:36 CST] [INFO] GetAllTimestamp biggestNew:{1667281485 1}, smallestNew:{1667281485 1}, biggestOld:{1667279683 1}, smallestOld:{1667279683 1}, MongoSource:[url[mongodb://root:***@192.168.0.23:30700,192.168.0.24:30701], name[rs0]], tsMap:map[rs0:{7160911711770247169 7160919451301314561}]
[2022/11/01 13:45:36 CST] [INFO] all node timestamp map: map[rs0:{7160911711770247169 7160919451301314561}] CheckpointStartPosition:{1 0}
[2022/11/01 13:45:36 CST] [INFO] New session to mongodb://root:***@192.168.0.23:30700,192.168.0.24:30701 successfully
[2022/11/01 13:45:36 CST] [INFO] rs0 Load exist checkpoint. content {"name":"rs0","ckpt":7160913867843829761,"version":2,"fetch_method":"oplog","oplog_disk_queue":"","oplog_disk_queue_apply_finish_ts":0}
[2022/11/01 13:45:36 CST] [INFO] rs0 checkpoint using mongod/replica_set: {"name":"rs0","ckpt":7160913867843829761,"version":2,"fetch_method":"oplog","oplog_disk_queue":"","oplog_disk_queue_apply_finish_ts":0}, ckptRemote set? [true]
[2022/11/01 13:45:36 CST] [INFO] sync mode run incr
[2022/11/01 13:45:36 CST] [INFO] start running with mode[incr], fullBeginTs[0[0, 0]]
[2022/11/01 13:45:36 CST] [INFO] start incr replication
[2022/11/01 13:45:36 CST] [INFO] RealSourceIncrSync[0]: url[mongodb://root:***@192.168.0.23:30700,192.168.0.24:30701], name[rs0], startTimestamp[7160913867843829761]
[2022/11/01 13:45:36 CST] [INFO] New session to mongodb://root:***@192.168.0.23:30702,192.168.0.24:30703 successfully

    • 1
    • 2
    • 3
    • 4
    • 5
    • 6
    • 7
    • 8
    • 9
    • 10
    • 11
    • 12
    • 13
    • 14
    • 15
    • 16
    • 17
    • 18
    • 19
    • 20
    • 21
    • 22
    • 23
    • 24
    • 25
    • 26
    • 27
    • 28
    • 29
    • 30
    • 31
    • 32
    • 33
    • 34
    • 35

    监控MongoShake状态，增量数据同步开始后，可以再开启一个命令行窗口，通过如下命令来监控MongoShake。

    cd /opt/mongoshake && ./mongoshake-stat --port=9100

    • 1

    说明： mongoshake-stat是一个Python脚本，执行之前请先安装Python 2.7版本。

    监控输出示例。

    [root@Ops-repo ~]# cd /opt/mongo-shake/
[root@Ops-repo mongo-shake]# ./mongoshake-stat --port=9100
|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|
|        log_size_avg |        log_size_max |        logs_get/sec |       logs_repl/sec |    logs_success/sec |            lsn.time |        lsn_ack.time |       lsn_ckpt.time |            now.time |             replset |             tps/sec |
|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|
|             331.00B |             339.00B |                none |                none |                none | 1970-01-01 08:00:00 | 1970-01-01 08:00:00 | 1970-01-01 08:00:00 | 2022-11-01 13:21:22 |                 rs0 |                none |
|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|
|             331.00B |             339.00B |                   0 |                   0 |                   0 | 1970-01-01 08:00:00 | 1970-01-01 08:00:00 | 1970-01-01 08:00:00 | 2022-11-01 13:21:23 |                 rs0 |                   0 |
|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|
|             331.00B |             339.00B |                   0 |                   0 |                   0 | 1970-01-01 08:00:00 | 1970-01-01 08:00:00 | 1970-01-01 08:00:00 | 2022-11-01 13:21:24 |                 rs0 |                   0 |
|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|
|             217.00B |             339.00B |                   1 |                   0 |                   0 | 1970-01-01 08:00:00 | 1970-01-01 08:00:00 | 1970-01-01 08:00:00 | 2022-11-01 13:21:25 |                 rs0 |                   0 |

    • 1
    • 2
    • 3
    • 4
    • 5
    • 6
    • 7
    • 8
    • 9
    • 10
    • 11
    • 12

    连接目标端副本集验证数据同步成功

    [root@Ops-repo bin]# mongo --host "192.168.0.23:30702,192.168.0.24:30703" \
>   --authenticationDatabase admin -u root -p o72ntfVFVL
MongoDB shell version v4.4.4
connecting to: mongodb://192.168.0.23:30702,192.168.0.24:30703/admin?authSource=admin&compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("18118dec-ac80-401b-bba5-2c8a80bd0066") }
MongoDB server version: 6.0.2
WARNING: shell and server versions do not match
---
The server generated these startup warnings when booting: 
        2022-11-01T05:18:05.971+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem
        2022-11-01T05:18:07.811+00:00: vm.max_map_count is too low
        2022-11-01T05:18:07.811+00:00:         currentValue: 262144
        2022-11-01T05:18:07.811+00:00:         recommendedMinimum: 1677720
        2022-11-01T05:18:07.811+00:00:         maxConns: 838860
---
---
        Enable MongoDB's free cloud-based monitoring service, which will then receive and display
        metrics about your deployment (disk utilization, CPU, operation statistics, etc).

        The monitoring data will be available on a MongoDB website with a unique URL accessible to you
        and anyone you share the URL with. MongoDB may use this information to make product
        improvements and to suggest MongoDB products and deployment options to you.

        To enable free monitoring, run the following command: db.enableFreeMonitoring()
        To permanently disable this reminder, run the following command: db.disableFreeMonitoring()
---
rs0:PRIMARY> 
rs0:PRIMARY> show dbs;
admin   0.000GB
config  0.000GB
local   0.001GB
testdb  0.000GB
rs0:PRIMARY> use testdb;
switched to db testdb
rs0:PRIMARY> show tables;
users
rs0:PRIMARY> show collections;
users
rs0:PRIMARY> 

    • 1
    • 2
    • 3
    • 4
    • 5
    • 6
    • 7
    • 8
    • 9
    • 10
    • 11
    • 12
    • 13
    • 14
    • 15
    • 16
    • 17
    • 18
    • 19
    • 20
    • 21
    • 22
    • 23
    • 24
    • 25
    • 26
    • 27
    • 28
    • 29
    • 30
    • 31
    • 32
    • 33
    • 34
    • 35
    • 36
    • 37
    • 38
    • 39
  • 相关阅读:
    第七章：最新版零基础学习 PYTHON 教程—Python 列表（第八节 -在 Python 中获取列表作为用户的输入）
    java毕业设计-基于springboot+vue的在线婚纱定制系统设计与实现,基于java的在线婚纱摄影预定系统，基于web的婚纱影楼管理系统设计，基于web婚纱影楼管理系统设计（附源码和配套资料）
    JavaScript-ES6-基础语法
    基础复习——为activity补充活动信息——利用资源文件配置字符串——利用元数据传递配置信息——给页面注册快捷方式...
    Python+Appium+Pytest+Allure自动化测试框架-代码篇
    GO通道：无缓冲通道与缓冲通道
    机器学习随笔（1）——pandas.DataFrame和数据清洗
    【安装部署】DataEase 版本回退
    1.9 动态解密ShellCode反弹
    nginx.4——正向代理和反向代理（七层代理和四层代理）
  • 原文地址：https://blog.csdn.net/networken/article/details/127632125