• s3cmd常用命令


    记录:319

    场景:在CentOS 7.9操作系统上,使用s3cmd操作Ceph分布式存储系统。操作包括上传、下载、检索和管理数据等。

    版本:

    操作系统:CentOS 7.9

    Ceph:Ceph version 13.2.10

    名词:

    Amazon S3,Amazon Simple Storage Service的简称,是一种对象存储服务,使用唯一性键值来存储任意数量的对象。

    S3cmd,是一个免费的命令行工具和客户端,用于在Amazon S3和其他使用S3协议的云存储服务提供商中上传、下载、检索和管理数据。比如,Ceph支持S3协议。

    Ceph,一个开源的分布式存储系统。

    bucket,在Ceph的对象存储模式中,bucket存放格式是:s3://BUCKET,相当于根目录下的一级目录;对象在bucket中存放格式:s3://BUCKET/OBJECT。

    bucket,直译:桶。

    1.基础环境

    1.1 Ceph集群(服务端)

    Ceph集群中已部署rgw组件且服务已启动,服务名称:radosgw。

    1.2 Ceph的客户端

    本例在集群之外的机器app166中部署客户端。

    (1)安装Ceph的客户端

    安装命令:yum install -y ceph-common

    (2)生成客户端秘钥

    生成命令:

    radosgw-admin user create --uid="hangzhou" --display-name="hangzhou"

    解析:radosgw-admin user create命令,创建用户;--uid,指定用户名;--display-name,指定显示名称。

    记录下关键信息:

    1. ......
    2. "user": "hangzhou",
    3. "access_key": "0DEWPO0TLWQWVN9ZU4LW",
    4. "secret_key": "D3ZXtteflGqnPTcjFRlQEuTO4qtbiTFIQHl2pBEZ"
    5. ......

    1.3 安装s3cmd客户端

    (1)安装s3cmd

    安装s3cmd命令:yum install -y s3cmd

    安装辅助包:yum install -y python-pip

    (2)配置s3cmd

    配置命令:vi /root/.s3cfg

    修改内容:

    1. [default]
    2. access_key=0DEWPO0TLWQWVN9ZU4LW
    3. secret_key=D3ZXtteflGqnPTcjFRlQEuTO4qtbiTFIQHl2pBEZ
    4. host_base=192.168.19.161:7480
    5. host_bucket=192.168.19.161:7480/%(bucket)
    6. cloudfront_host=192.168.19.161:7480
    7. use_https=False

    解析:192.168.19.161:7480,是部署Ceph的radosgw的主机和端口;access_key和secret_key是使用radosgw-admin user create命令生成的。

    注意:配置文件/root/.s3cfg中,建议不要写额外无用信息。比如#注释。

    2.s3cmd常用命令

    (1)ps帮助命令

    命令:s3cmd --help

    功能:查看s3cmd支持全部命令和选项,在实际工作中,查看这个手册应该是必备之选。

    (2)配置s3cmd

    命令:s3cmd --configure

    功能:一个交互式配置工具。

    (3)创建bucket

    命令:s3cmd mb s3://hangzhou

    功能:创建一个bucket,名称:hangzhou。bucket名称唯一,不能重复。

    (4)删除空bucket

    命令:s3cmd rb s3://hangzhou

    功能:只能删除空桶,如果桶有内容需先删内容,再删除桶。

    (5)查看bucket

    命令:s3cmd ls

    功能:查看桶列表。

    命令:s3cmd ls s3://hangzhou/

    功能:查看指定桶下的对象。

    (6)查看bucket包括内容

    命令:s3cmd la

    功能:查看s3://下的桶列表,包括桶里面的对象。

    (7)上传文件到bucket

    命令:s3cmd put /home/jdk-8u281-linux-x64.tar.gz s3://hangzhou

    功能:上传文件到ceph的桶中存储。

    (8)从bucket下文件

    命令:s3cmd get s3://hangzhou/jdk-8u281-linux-x64.tar.gz

    功能:从ceph的桶中下载文件到本地。

    (9)删除bucket中的文件

    命令:s3cmd del s3://hangzhou/jdk-8u281-linux-x64.tar.gz

    命令:s3cmd rm s3://hangzhou/jdk-8u281-linux-x64.tar.gz

    功能:删除桶中文件。

    (10)查看bucket已使用空间

    命令:s3cmd du -H s3://hangzhou

    功能:查看桶中对象,占用空间大小。

    (11)获取bucket的信息

    命令:s3cmd info s3://hangzhou

    功能:查看桶信息。

    命令:s3cmd info s3://hangzhou/jdk-8u281-linux-x64.tar.gz

    功能:查看桶中对象信息。

    (12)复制bucket中文件

    命令:s3cmd cp s3://hangzhou/jdk-8u281-linux-x64.tar.gz s3://hangzhou_new

    功能:把文件从一个桶复制到另一个桶。

    (13)移动bucket中文件

    命令:s3cmd mv s3://hangzhou/jdk-8u281-linux-x64.tar.gz s3://hangzhou_new

    功能:把文件从一个桶移动到另一个桶。

    (14)本地目录下文件同步bucket

    命令:s3cmd sync /home/apps/software/ s3://hangzhou

    功能:指定目录下文件,同步到ceph的桶中。

    (15)对比指定目录下文件和bucket中文件差异

    命令:s3cmd sync --dry-run /home/apps/software/ s3://hangzhou

    功能:比对本地目录下文件和bucket文件差异。

    (16)对比指定目录下文件和bucket中文件并删除桶中差异文件

    命令:s3cmd sync --delete-removed /home/apps/software/ s3://hangzhou_new

    功能:对比指定目录下文件和bucket中文件,删除指定目录没有但是在桶中有的文件;上传指定目录有的文件但是桶中没有的文件。

    3.帮助命令详细列表

    命令:s3cmd --help

    查看s3cmd支持全部命令和选项,在实际工作中,查看这个手册应该是必备之选。

    1. Usage: s3cmd [options] COMMAND [parameters]
    2. S3cmd is a tool for managing objects in Amazon S3 storage. It allows for
    3. making and removing "buckets" and uploading, downloading and removing
    4. "objects" from these buckets.
    5. Options:
    6. -h, --help show this help message and exit
    7. --configure Invoke interactive (re)configuration tool. Optionally
    8. use as '--configure s3://some-bucket' to test access
    9. to a specific bucket instead of attempting to list
    10. them all.
    11. -c FILE, --config=FILE
    12. Config file name. Defaults to $HOME/.s3cfg
    13. --dump-config Dump current configuration after parsing config files
    14. and command line options and exit.
    15. --access_key=ACCESS_KEY
    16. AWS Access Key
    17. --secret_key=SECRET_KEY
    18. AWS Secret Key
    19. --access_token=ACCESS_TOKEN
    20. AWS Access Token
    21. -n, --dry-run Only show what should be uploaded or downloaded but
    22. don't actually do it. May still perform S3 requests to
    23. get bucket listings and other information though (only
    24. for file transfer commands)
    25. -s, --ssl Use HTTPS connection when communicating with S3.
    26. (default)
    27. --no-ssl Don't use HTTPS.
    28. -e, --encrypt Encrypt files before uploading to S3.
    29. --no-encrypt Don't encrypt files.
    30. -f, --force Force overwrite and other dangerous operations.
    31. --continue Continue getting a partially downloaded file (only for
    32. [get] command).
    33. --continue-put Continue uploading partially uploaded files or
    34. multipart upload parts. Restarts parts/files that
    35. don't have matching size and md5. Skips files/parts
    36. that do. Note: md5sum checks are not always
    37. sufficient to check (part) file equality. Enable this
    38. at your own risk.
    39. --upload-id=UPLOAD_ID
    40. UploadId for Multipart Upload, in case you want
    41. continue an existing upload (equivalent to --continue-
    42. put) and there are multiple partial uploads. Use
    43. s3cmd multipart [URI] to see what UploadIds are
    44. associated with the given URI.
    45. --skip-existing Skip over files that exist at the destination (only
    46. for [get] and [sync] commands).
    47. -r, --recursive Recursive upload, download or removal.
    48. --check-md5 Check MD5 sums when comparing files for [sync].
    49. (default)
    50. --no-check-md5 Do not check MD5 sums when comparing files for [sync].
    51. Only size will be compared. May significantly speed up
    52. transfer but may also miss some changed files.
    53. -P, --acl-public Store objects with ACL allowing read for anyone.
    54. --acl-private Store objects with default ACL allowing access for you
    55. only.
    56. --acl-grant=PERMISSION:EMAIL or USER_CANONICAL_ID
    57. Grant stated permission to a given amazon user.
    58. Permission is one of: read, write, read_acp,
    59. write_acp, full_control, all
    60. --acl-revoke=PERMISSION:USER_CANONICAL_ID
    61. Revoke stated permission for a given amazon user.
    62. Permission is one of: read, write, read_acp,
    63. write_acp, full_control, all
    64. -D NUM, --restore-days=NUM
    65. Number of days to keep restored file available (only
    66. for 'restore' command). Default is 1 day.
    67. --restore-priority=RESTORE_PRIORITY
    68. Priority for restoring files from S3 Glacier (only for
    69. 'restore' command). Choices available: bulk, standard,
    70. expedited
    71. --delete-removed Delete destination objects with no corresponding
    72. source file [sync]
    73. --no-delete-removed Don't delete destination objects [sync]
    74. --delete-after Perform deletes AFTER new uploads when delete-removed
    75. is enabled [sync]
    76. --delay-updates *OBSOLETE* Put all updated files into place at end
    77. [sync]
    78. --max-delete=NUM Do not delete more than NUM files. [del] and [sync]
    79. --limit=NUM Limit number of objects returned in the response body
    80. (only for [ls] and [la] commands)
    81. --add-destination=ADDITIONAL_DESTINATIONS
    82. Additional destination for parallel uploads, in
    83. addition to last arg. May be repeated.
    84. --delete-after-fetch Delete remote objects after fetching to local file
    85. (only for [get] and [sync] commands).
    86. -p, --preserve Preserve filesystem attributes (mode, ownership,
    87. timestamps). Default for [sync] command.
    88. --no-preserve Don't store FS attributes
    89. --exclude=GLOB Filenames and paths matching GLOB will be excluded
    90. from sync
    91. --exclude-from=FILE Read --exclude GLOBs from FILE
    92. --rexclude=REGEXP Filenames and paths matching REGEXP (regular
    93. expression) will be excluded from sync
    94. --rexclude-from=FILE Read --rexclude REGEXPs from FILE
    95. --include=GLOB Filenames and paths matching GLOB will be included
    96. even if previously excluded by one of
    97. --(r)exclude(-from) patterns
    98. --include-from=FILE Read --include GLOBs from FILE
    99. --rinclude=REGEXP Same as --include but uses REGEXP (regular expression)
    100. instead of GLOB
    101. --rinclude-from=FILE Read --rinclude REGEXPs from FILE
    102. --files-from=FILE Read list of source-file names from FILE. Use - to
    103. read from stdin.
    104. --region=REGION, --bucket-location=REGION
    105. Region to create bucket in. As of now the regions are:
    106. us-east-1, us-west-1, us-west-2, eu-west-1, eu-
    107. central-1, ap-northeast-1, ap-southeast-1, ap-
    108. southeast-2, sa-east-1
    109. --host=HOSTNAME HOSTNAME:PORT for S3 endpoint (default:
    110. s3.amazonaws.com, alternatives such as s3-eu-
    111. west-1.amazonaws.com). You should also set --host-
    112. bucket.
    113. --host-bucket=HOST_BUCKET
    114. DNS-style bucket+hostname:port template for accessing
    115. a bucket (default: %(bucket)s.s3.amazonaws.com)
    116. --reduced-redundancy, --rr
    117. Store object with 'Reduced redundancy'. Lower per-GB
    118. price. [put, cp, mv]
    119. --no-reduced-redundancy, --no-rr
    120. Store object without 'Reduced redundancy'. Higher per-
    121. GB price. [put, cp, mv]
    122. --storage-class=CLASS
    123. Store object with specified CLASS (STANDARD,
    124. STANDARD_IA, ONEZONE_IA, INTELLIGENT_TIERING, GLACIER
    125. or DEEP_ARCHIVE). [put, cp, mv]
    126. --access-logging-target-prefix=LOG_TARGET_PREFIX
    127. Target prefix for access logs (S3 URI) (for [cfmodify]
    128. and [accesslog] commands)
    129. --no-access-logging Disable access logging (for [cfmodify] and [accesslog]
    130. commands)
    131. --default-mime-type=DEFAULT_MIME_TYPE
    132. Default MIME-type for stored objects. Application
    133. default is binary/octet-stream.
    134. -M, --guess-mime-type
    135. Guess MIME-type of files by their extension or mime
    136. magic. Fall back to default MIME-Type as specified by
    137. --default-mime-type option
    138. --no-guess-mime-type Don't guess MIME-type and use the default type
    139. instead.
    140. --no-mime-magic Don't use mime magic when guessing MIME-type.
    141. -m MIME/TYPE, --mime-type=MIME/TYPE
    142. Force MIME-type. Override both --default-mime-type and
    143. --guess-mime-type.
    144. --add-header=NAME:VALUE
    145. Add a given HTTP header to the upload request. Can be
    146. used multiple times. For instance set 'Expires' or
    147. 'Cache-Control' headers (or both) using this option.
    148. --remove-header=NAME Remove a given HTTP header. Can be used multiple
    149. times. For instance, remove 'Expires' or 'Cache-
    150. Control' headers (or both) using this option. [modify]
    151. --server-side-encryption
    152. Specifies that server-side encryption will be used
    153. when putting objects. [put, sync, cp, modify]
    154. --server-side-encryption-kms-id=KMS_KEY
    155. Specifies the key id used for server-side encryption
    156. with AWS KMS-Managed Keys (SSE-KMS) when putting
    157. objects. [put, sync, cp, modify]
    158. --encoding=ENCODING Override autodetected terminal and filesystem encoding
    159. (character set). Autodetected: UTF-8
    160. --add-encoding-exts=EXTENSIONs
    161. Add encoding to these comma delimited extensions i.e.
    162. (css,js,html) when uploading to S3 )
    163. --verbatim Use the S3 name as given on the command line. No pre-
    164. processing, encoding, etc. Use with caution!
    165. --disable-multipart Disable multipart upload on files bigger than
    166. --multipart-chunk-size-mb
    167. --multipart-chunk-size-mb=SIZE
    168. Size of each chunk of a multipart upload. Files bigger
    169. than SIZE are automatically uploaded as multithreaded-
    170. multipart, smaller files are uploaded using the
    171. traditional method. SIZE is in Mega-Bytes, default
    172. chunk size is 15MB, minimum allowed chunk size is 5MB,
    173. maximum is 5GB.
    174. --list-md5 Include MD5 sums in bucket listings (only for 'ls'
    175. command).
    176. --list-allow-unordered
    177. Not an AWS standard. Allow the listing results to be
    178. returned in unsorted order. This may be faster when
    179. listing very large buckets.
    180. -H, --human-readable-sizes
    181. Print sizes in human readable form (eg 1kB instead of
    182. 1234).
    183. --ws-index=WEBSITE_INDEX
    184. Name of index-document (only for [ws-create] command)
    185. --ws-error=WEBSITE_ERROR
    186. Name of error-document (only for [ws-create] command)
    187. --expiry-date=EXPIRY_DATE
    188. Indicates when the expiration rule takes effect. (only
    189. for [expire] command)
    190. --expiry-days=EXPIRY_DAYS
    191. Indicates the number of days after object creation the
    192. expiration rule takes effect. (only for [expire]
    193. command)
    194. --expiry-prefix=EXPIRY_PREFIX
    195. Identifying one or more objects with the prefix to
    196. which the expiration rule applies. (only for [expire]
    197. command)
    198. --progress Display progress meter (default on TTY).
    199. --no-progress Don't display progress meter (default on non-TTY).
    200. --stats Give some file-transfer stats.
    201. --enable Enable given CloudFront distribution (only for
    202. [cfmodify] command)
    203. --disable Disable given CloudFront distribution (only for
    204. [cfmodify] command)
    205. --cf-invalidate Invalidate the uploaded filed in CloudFront. Also see
    206. [cfinval] command.
    207. --cf-invalidate-default-index
    208. When using Custom Origin and S3 static website,
    209. invalidate the default index file.
    210. --cf-no-invalidate-default-index-root
    211. When using Custom Origin and S3 static website, don't
    212. invalidate the path to the default index file.
    213. --cf-add-cname=CNAME Add given CNAME to a CloudFront distribution (only for
    214. [cfcreate] and [cfmodify] commands)
    215. --cf-remove-cname=CNAME
    216. Remove given CNAME from a CloudFront distribution
    217. (only for [cfmodify] command)
    218. --cf-comment=COMMENT Set COMMENT for a given CloudFront distribution (only
    219. for [cfcreate] and [cfmodify] commands)
    220. --cf-default-root-object=DEFAULT_ROOT_OBJECT
    221. Set the default root object to return when no object
    222. is specified in the URL. Use a relative path, i.e.
    223. default/index.html instead of /default/index.html or
    224. s3://bucket/default/index.html (only for [cfcreate]
    225. and [cfmodify] commands)
    226. -v, --verbose Enable verbose output.
    227. -d, --debug Enable debug output.
    228. --version Show s3cmd version (2.3.0) and exit.
    229. -F, --follow-symlinks
    230. Follow symbolic links as if they are regular files
    231. --cache-file=FILE Cache FILE containing local source MD5 values
    232. -q, --quiet Silence output on stdout
    233. --ca-certs=CA_CERTS_FILE
    234. Path to SSL CA certificate FILE (instead of system
    235. default)
    236. --ssl-cert=SSL_CLIENT_CERT_FILE
    237. Path to client own SSL certificate CRT_FILE
    238. --ssl-key=SSL_CLIENT_KEY_FILE
    239. Path to client own SSL certificate private key
    240. KEY_FILE
    241. --check-certificate Check SSL certificate validity
    242. --no-check-certificate
    243. Do not check SSL certificate validity
    244. --check-hostname Check SSL certificate hostname validity
    245. --no-check-hostname Do not check SSL certificate hostname validity
    246. --signature-v2 Use AWS Signature version 2 instead of newer signature
    247. methods. Helpful for S3-like systems that don't have
    248. AWS Signature v4 yet.
    249. --limit-rate=LIMITRATE
    250. Limit the upload or download speed to amount bytes per
    251. second. Amount may be expressed in bytes, kilobytes
    252. with the k suffix, or megabytes with the m suffix
    253. --no-connection-pooling
    254. Disable connection re-use
    255. --requester-pays Set the REQUESTER PAYS flag for operations
    256. -l, --long-listing Produce long listing [ls]
    257. --stop-on-error stop if error in transfer
    258. --content-disposition=CONTENT_DISPOSITION
    259. Provide a Content-Disposition for signed URLs, e.g.,
    260. "inline; filename=myvideo.mp4"
    261. --content-type=CONTENT_TYPE
    262. Provide a Content-Type for signed URLs, e.g.,
    263. "video/mp4"
    264. Commands:
    265. Make bucket
    266. s3cmd mb s3://BUCKET
    267. Remove bucket
    268. s3cmd rb s3://BUCKET
    269. List objects or buckets
    270. s3cmd ls [s3://BUCKET[/PREFIX]]
    271. List all object in all buckets
    272. s3cmd la
    273. Put file into bucket
    274. s3cmd put FILE [FILE...] s3://BUCKET[/PREFIX]
    275. Get file from bucket
    276. s3cmd get s3://BUCKET/OBJECT LOCAL_FILE
    277. Delete file from bucket
    278. s3cmd del s3://BUCKET/OBJECT
    279. Delete file from bucket (alias for del)
    280. s3cmd rm s3://BUCKET/OBJECT
    281. Restore file from Glacier storage
    282. s3cmd restore s3://BUCKET/OBJECT
    283. Synchronize a directory tree to S3 (checks files freshness using size and md5 checksum, unless overridden by options, see below)
    284. s3cmd sync LOCAL_DIR s3://BUCKET[/PREFIX] or s3://BUCKET[/PREFIX] LOCAL_DIR or s3://BUCKET[/PREFIX] s3://BUCKET[/PREFIX]
    285. Disk usage by buckets
    286. s3cmd du [s3://BUCKET[/PREFIX]]
    287. Get various information about Buckets or Files
    288. s3cmd info s3://BUCKET[/OBJECT]
    289. Copy object
    290. s3cmd cp s3://BUCKET1/OBJECT1 s3://BUCKET2[/OBJECT2]
    291. Modify object metadata
    292. s3cmd modify s3://BUCKET1/OBJECT
    293. Move object
    294. s3cmd mv s3://BUCKET1/OBJECT1 s3://BUCKET2[/OBJECT2]
    295. Modify Access control list for Bucket or Files
    296. s3cmd setacl s3://BUCKET[/OBJECT]
    297. Modify Bucket Policy
    298. s3cmd setpolicy FILE s3://BUCKET
    299. Delete Bucket Policy
    300. s3cmd delpolicy s3://BUCKET
    301. Modify Bucket CORS
    302. s3cmd setcors FILE s3://BUCKET
    303. Delete Bucket CORS
    304. s3cmd delcors s3://BUCKET
    305. Modify Bucket Requester Pays policy
    306. s3cmd payer s3://BUCKET
    307. Show multipart uploads
    308. s3cmd multipart s3://BUCKET [Id]
    309. Abort a multipart upload
    310. s3cmd abortmp s3://BUCKET/OBJECT Id
    311. List parts of a multipart upload
    312. s3cmd listmp s3://BUCKET/OBJECT Id
    313. Enable/disable bucket access logging
    314. s3cmd accesslog s3://BUCKET
    315. Sign arbitrary string using the secret key
    316. s3cmd sign STRING-TO-SIGN
    317. Sign an S3 URL to provide limited public access with expiry
    318. s3cmd signurl s3://BUCKET/OBJECT
    319. Fix invalid file names in a bucket
    320. s3cmd fixbucket s3://BUCKET[/PREFIX]
    321. Create Website from bucket
    322. s3cmd ws-create s3://BUCKET
    323. Delete Website
    324. s3cmd ws-delete s3://BUCKET
    325. Info about Website
    326. s3cmd ws-info s3://BUCKET
    327. Set or delete expiration rule for the bucket
    328. s3cmd expire s3://BUCKET
    329. Upload a lifecycle policy for the bucket
    330. s3cmd setlifecycle FILE s3://BUCKET
    331. Get a lifecycle policy for the bucket
    332. s3cmd getlifecycle s3://BUCKET
    333. Remove a lifecycle policy for the bucket
    334. s3cmd dellifecycle s3://BUCKET
    335. Upload a notification policy for the bucket
    336. s3cmd setnotification FILE s3://BUCKET
    337. Get a notification policy for the bucket
    338. s3cmd getnotification s3://BUCKET
    339. Remove a notification policy for the bucket
    340. s3cmd delnotification s3://BUCKET
    341. List CloudFront distribution points
    342. s3cmd cflist
    343. Display CloudFront distribution point parameters
    344. s3cmd cfinfo [cf://DIST_ID]
    345. Create CloudFront distribution point
    346. s3cmd cfcreate s3://BUCKET
    347. Delete CloudFront distribution point
    348. s3cmd cfdelete cf://DIST_ID
    349. Change CloudFront distribution point parameters
    350. s3cmd cfmodify cf://DIST_ID
    351. Display CloudFront invalidation request(s) status
    352. s3cmd cfinvalinfo cf://DIST_ID[/INVAL_ID]
    353. For more information, updates and news, visit the s3cmd website:
    354. http://s3tools.org

    以上,感谢。

    2022年11月17日

  • 相关阅读:
    vue问题01:路由鉴权跳转页面,vuex+sessionStorage中的token不一致,刷新页面数据丢失
    MySQL的常用优化方案
    mpls详解
    java计算机毕业设计在线校园超市系统源代码+系统+数据库+lw文档
    设计模式 -- 适配器模式(Adapter Pattern)
    ElasticSearch 文档数据导入导出及两个ElasticSearch互相导入操作
    带你吃透Servlet技术(一)
    Git 安装和基础命令、IDEA 基础操作
    MAUI Blazor 权限经验分享 (定位,使用相机)
    JUnit5的条件测试、嵌套测试、重复测试
  • 原文地址:https://blog.csdn.net/zhangbeizhen18/article/details/127913300