• 15.镜像安全-镜像加密

    Overview

    容器安全可以归纳分类如下：

    • 1、容器(+镜像)是否有病毒——代表项目：clair, IBM Vulnerability Advisor
    • 2、容器(+镜像)是否被篡改——代表项目：Notary
    • 3、容器是否隔离(container runtime security)——代表项目：runc
    • 4、容器(+镜像)是否保密，也即镜像是否被目标用户运行——代表项目：containerd, cri-o

    本文主要讨论第四点——镜像保密性。社区目前对于镜像保密性的工作主要由IBM公司的Brandon Lum和Harshal Patil主导进行。由于镜像加密特性会改动OCI标准中的镜像格式部分：the Image Specification image-spec，因此在详细介绍镜像加密原理之前有必要先了解一下OCI

    OCI

    OCI(Open Container Initiative)由 Docker，CoreOS以及容器行业中的其他领导者在2015年6月启动，致力于围绕容器格式和运行时创建开放的行业标准，主要包含两部分：

    • 1、the Runtime Specification (runtime-spec)——负责容器运行时，contains technologies such as namespaces, cgroups, Linux capabilities, as well as SELinux, AppArmor, and Seccomp profiles
    • 2、the Image Specification (image-spec)——负责容器镜像标准，主要规范镜像存储格式

    社区对于OCI规范的实现情况如下：

    支持OCI Runtime Spec&Image Spec的项目如下：

    为此containerd, cri-o, docker以及rkt也被广义地称为container runtime(High-Level Container Runtimes)，如图：

    详情见An Introduction to Container Runtimes

    Encrypted Container Image Principle

    由于镜像加密是在OCI镜像格式上的扩展，我们先介绍现存镜像格式，如下：

    1. {
    2.   "schemaVersion": 2,
    3.   "config": {
    4.     "mediaType": "application/vnd.oci.image.config.v1+json",
    5.     "size": 7023,
    6.     "digest": "sha256:b5b2b2c507a0944348e0303114d8d93aaaa081732b86451d9bce1f432a537bc7"
    7.   },
    8.   "layers": [
    9.     {
    10.       "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
    11.       "size": 32654,
    12.       "digest": "sha256:9834876dcfb05cb167a5c24953eba58c4ac89b1adf57f28f2f9d09af107ee8f0"
    13.     },
    14.     {
    15.       "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
    16.       "size": 16724,
    17.       "digest": "sha256:3c3a4604a545cdc127456d94e421cd355bca5b528f4a9c1905b15da2eb4a4c6b"
    18.     },
    19.     {
    20.       "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
    21.       "size": 73109,
    22.       "digest": "sha256:ec4b8955958665577945c89419d1af06b5f7636b4ac3da7f12184802ad867736"
    23.     }
    24.   ],
    25.   "annotations": {
    26.     "com.example.key1": "value1",
    27.     "com.example.key2": "value2"
    28.   }
    29. }

    Unlike the image index, which contains information about a set of images that can span a variety of architectures and operating systems, an image manifest provides a configuration and set of layers for a single container image for a specific architecture and operating system.

    对该格式说明如下(refers to Image Manifest Property Descriptions)：

    而基于Proposal: New Mediatype - Container Image Encryption的讨论，镜像加密原理图(draft of modifications to the OCI spec on Encrypted Container Images)如下：

    Encrypted container images are based on the OCI image spec. The changes to the spec is the adding of the encrypted layer mediatype. An image manifest contains some metadata and a list of layers. We introduce a new layer mediatype suffix +encrypted to represent an encrypted layer. For example, a regular layer with mediatype ‘application/vnd.oci.image.layer.v1.tar’ when encrypted would be ‘application/vnd.oci.image.layer.v1.tar+encrypted’. Because there is some metadata involved with the encryption, an encrypted layer will also contain several annotations with the prefix org.opencontainers.image.enc.

    也即在原来格式的基础上添加了一个mediaType类型，表示数据文件被加密；同时在annotation中添加具体加密相关信息

    例如，没加密之前数据如下：

    1. "layers":[
    2.   {
    3.     "mediaType":"application/vnd.oci.image.layer.v1.tar+gzip",
    4.     "digest":"sha256:7c9d20b9b6cda1c58bc4f9d6c401386786f584437abbe87e58910f8a9a15386b",
    5.     "size":760770
    6.   }
    7. ]

    加密之后数据如下：

    1. "layers":[
    2.   {
    3.     "mediaType":"application/vnd.oci.image.layer.v1.tar+gzip+encrypted",
    4.     "digest":"sha256:c72c69b36a886c268e0d7382a7c6d885271b6f0030ff022fda2b6346b2b274ba",
    5.     "size":760770,
    6.     "annotations": {
    7.       "org.opencontainers.image.enc.keys.jwe":"eyJwcm90ZWN0Z...",
    8.       "org.opencontainers.image.enc.pubopts":"eyJjaXBoZXIiOi..."
    9.     }
    10.   }
    11. ]

    另外，通过将加密粒度细化到镜像层，保留了原有镜像格式的layering and deduplication特性

    整个加密体系使用混合加密方案：

    • 1、使用对称加密算法(i.e. AES)加密镜像分层数据——主要为了提高加密解密速度
    • 2、使用非对称加密算法(eg: OpenPGP, JSON Web Encryption (JWE), and PKCS#7)首先用公钥加密上述对称加密的密钥，然后用私钥对加密后的密钥进行解密——主要为了提高密钥管理的灵活性

    流程图如下：

     

    大致可以归纳为如下步骤：

    • 1、产生对称加密密钥(i.e. LEK)，用于加密镜像分层
    • 2、利用非对称加密公钥加密LEK，产生Wrapped Key
    • 3、将加密后的镜像(包含Wrapped Key)上传到公开仓库，例如：Docker Hub or quay.io
    • 4、非对称加密私钥持有者下载该镜像，并用该私钥对该镜像进行解密并运行

    Encrypted Container Image Build ToolChain——containerd imgcrypt

    对于镜像加密工具，社区计划支持：docker CLI(To integrate into docker CLI, we are currently waiting on moby/moby#38043. Tracking with moby/buildkit#714), containerd imgcrypt, skopeo以及buildah，目前已经实现的有containerd imgcrypt和skopeo

    下面我们基于containerd imgcrypt对上述加密流程进行实操：

    1. # Step1: Install containerd imgcrypt
    2. $ git clone https://github.com/containerd/imgcrypt.git github.com/containerd/imgcrypt
    3. $ cd github.com/containerd/imgcrypt && make && make install
    4. install
    5.  
    6. # Step2: Setting up containerd
    7. $ wget https://github.com/containerd/containerd/releases/download/v1.3.0/containerd-1.3.0.linux-amd64.tar.gz
    8. $ tar -xzf containerd-1.3.0.linux-amd64.tar.gz
    9. $ ls bin/
    10. containerd  containerd-shim  containerd-shim-runc-v1  containerd-shim-runc-v2  containerd-stress  ctr
    11. $ cat <<EOF > config.toml
    12. disable_plugins = ["cri"]
    13. root = "/tmp/var/lib/containerd"
    14. state = "/tmp/run/containerd"
    15. [grpc]
    16.   address = "/tmp/run/containerd/mycontainerd.sock"
    17.   uid = 0
    18.   gid = 0
    19. [stream_processors]
    20.     [stream_processors."io.containerd.ocicrypt.decoder.v1.tar.gzip"]
    21.         accepts = ["application/vnd.oci.image.layer.v1.tar+gzip+encrypted"]
    22.         returns = "application/vnd.oci.image.layer.v1.tar+gzip"
    23.         path = "/usr/local/bin/ctd-decoder"
    24.     [stream_processors."io.containerd.ocicrypt.decoder.v1.tar"]
    25.         accepts = ["application/vnd.oci.image.layer.v1.tar+encrypted"]
    26.         returns = "application/vnd.oci.image.layer.v1.tar"
    27.         path = "/usr/local/bin/ctd-decoder"
    28. EOF
    29. $ bin/containerd -c config.toml
    30. [... truncated ...]
    31. INFO[2020-02-26T11:16:07.704084989+08:00] containerd successfully booted in 0.047800s
    32.  
    33. # Step3: Generate some RSA keys with openssl
    34. $ openssl genrsa -out mykey.pem
    35. Generating RSA private key, 2048 bit long modulus
    36. .......................................................................................................+++
    37. .............................+++
    38. e is 65537 (0x10001)
    39. $ openssl rsa -in mykey.pem -pubout -out mypubkey.pem
    40. writing RSA key
    41.  
    42. # Step4: Pulling an image
    43. $ chmod 0666 /tmp/run/containerd/containerd.sock
    44. $ CTR="/usr/local/bin/ctr-enc -a /tmp/run/containerd/containerd.sock"
    45. $ $CTR images pull --all-platforms docker.io/library/bash:latest
    46. [... truncated ...]
    47. elapsed: 8.7 s                                                                    total:  39.2 M (4.5 MiB/s)                                       
    48. unpacking linux/amd64 sha256:8193ca23afc8e1069bd3d982733df79c68dbcfcd66b4da6b5d65da85987dae2f...
    49. unpacking linux/arm/v6 sha256:8193ca23afc8e1069bd3d982733df79c68dbcfcd66b4da6b5d65da85987dae2f...
    50. unpacking linux/arm/v7 sha256:8193ca23afc8e1069bd3d982733df79c68dbcfcd66b4da6b5d65da85987dae2f...
    51. unpacking linux/arm64/v8 sha256:8193ca23afc8e1069bd3d982733df79c68dbcfcd66b4da6b5d65da85987dae2f...
    52. unpacking linux/386 sha256:8193ca23afc8e1069bd3d982733df79c68dbcfcd66b4da6b5d65da85987dae2f...
    53. unpacking linux/ppc64le sha256:8193ca23afc8e1069bd3d982733df79c68dbcfcd66b4da6b5d65da85987dae2f...
    54. unpacking linux/s390x sha256:8193ca23afc8e1069bd3d982733df79c68dbcfcd66b4da6b5d65da85987dae2f...
    55. done
    56. $ $CTR images list   
    57. REF                           TYPE                                                      DIGEST                                                                  SIZE    PLATFORMS                                                                                LABELS 
    58. docker.io/library/bash:latest application/vnd.docker.distribution.manifest.list.v2+json sha256:8193ca23afc8e1069bd3d982733df79c68dbcfcd66b4da6b5d65da85987dae2f 5.7 MiB linux/386,linux/amd64,linux/arm/v6,linux/arm/v7,linux/arm64/v8,linux/ppc64le,linux/s390x -   
    59. $ $CTR images layerinfo --platform linux/amd64 docker.io/library/bash:latest
    60.    #                                                                    DIGEST      PLATFORM      SIZE   ENCRYPTION   RECIPIENTS
    61.    0   sha256:c9b1b535fdd91a9855fb7f82348177e5f019329a58c53c47272962dd60f71fc9   linux/amd64   2802957                          
    62.    1   sha256:a3698fd3137d820dbb91b5f96e89af64b196dde4ab4c326dd1bd6291bbd771cf   linux/amd64   3186680                          
    63.    2   sha256:2538f4b330c99e67e96803569826ba62f7a3353d27f2a15bcf1e33d1814fa7ad   linux/amd64       340
    64.  
    65. # Step5: Encrypting the image
    66. $ $CTR images encrypt --recipient jwe:mypubkey.pem --platform linux/amd64 docker.io/library/bash:latest bash.enc:latest
    67. Encrypting docker.io/library/bash:latest to bash.enc:latest
    68. # The arguments are:
    69. # --recipient jwe:mypubkey.pem: This indicates that we want to encrypt the image using the public key mypubkey.pem that we just generated, and the prefix jwe: indicates that we want to use the encryption scheme JSON web encryption scheme for our encryption metadata.
    70. # --platform linux/amd64: Encrypt only the linux/amd64 image
    71. # docker.io/library/bash:latest - The image to encrypt
    72. # bash.enc:latest - The tag of the encrypted image to be created
    73. # Optional: it is possible to encrypt just certain layers of the image by using the --layer flag
    74. $ $CTR images list
    75. REF                           TYPE                                                      DIGEST                                                                  SIZE    PLATFORMS                                                                                LABELS 
    76. bash.enc:latest               application/vnd.oci.image.index.v1+json                   sha256:e1cea4c43a0d43bf3fdd6af66509668f6fe16e433085fcc7cf5b8a4ec48c80aa 5.7 MiB linux/386,linux/amd64,linux/arm/v6,linux/arm/v7,linux/arm64/v8,linux/ppc64le,linux/s390x -      
    77. docker.io/library/bash:latest application/vnd.docker.distribution.manifest.list.v2+json sha256:8193ca23afc8e1069bd3d982733df79c68dbcfcd66b4da6b5d65da85987dae2f 5.7 MiB linux/386,linux/amd64,linux/arm/v6,linux/arm/v7,linux/arm64/v8,linux/ppc64le,linux/s390x -   
    78. # We can observe the layerinfo to show the encryption
    79. $ $CTR images layerinfo --platform linux/amd64 bash.enc:latest
    80.    #                                                                    DIGEST      PLATFORM      SIZE   ENCRYPTION   RECIPIENTS
    81.    0   sha256:375abe9071e498e57f1cfcf349266948c3760bfb136e051c22a532d38a020e74   linux/amd64   2802957          jwe        [jwe]
    82.    1   sha256:3298803dec193fdd561ea27d0b1469e3f0b5b796962f543f3bcc57cd12ebdf5d   linux/amd64   3186680          jwe        [jwe]
    83.    2   sha256:bdbc04a60398d706a06e5ae88a4cdbfef770c9fe1b12e31fcf5a50532dafad87   linux/amd64       340          jwe        [jwe]
    84. # Strange enough, the layers 'SIZE' of 'bash.enc:latest' is same with the one of 'docker.io/library/bash:latest'
    85. $ $CTR images layerinfo --platform linux/amd64 docker.io/library/bash:latest
    86.    #                                                                    DIGEST      PLATFORM      SIZE   ENCRYPTION   RECIPIENTS
    87.    0   sha256:c9b1b535fdd91a9855fb7f82348177e5f019329a58c53c47272962dd60f71fc9   linux/amd64   2802957                          
    88.    1   sha256:a3698fd3137d820dbb91b5f96e89af64b196dde4ab4c326dd1bd6291bbd771cf   linux/amd64   3186680                          
    89.    2   sha256:2538f4b330c99e67e96803569826ba62f7a3353d27f2a15bcf1e33d1814fa7ad   linux/amd64       340
    90.  
    91. # Compare other platforms
    92. $ $CTR images layerinfo --platform linux/arm64/v8 bash.enc:latest
    93.    #                                                                    DIGEST         PLATFORM      SIZE   ENCRYPTION   RECIPIENTS
    94.    0   sha256:8fa90b21c985a6fcfff966bdfbde81cdd088de0aa8af38110057f6ac408f4408   linux/arm64/v8   2723075                          
    95.    1   sha256:4d89eb6b628bc985ffcdcb939bf05a16194be6aed18c2848e1424d01ca0012fd   linux/arm64/v8   3193678                          
    96.    2   sha256:4818fb358d8f64c071010cb05fe59285888a36edbe1a45a5eed80576b08d312c   linux/arm64/v8       344                          
    97. $ $CTR images layerinfo --platform linux/arm64/v8 docker.io/library/bash:latest
    98.    #                                                                    DIGEST         PLATFORM      SIZE   ENCRYPTION   RECIPIENTS
    99.    0   sha256:8fa90b21c985a6fcfff966bdfbde81cdd088de0aa8af38110057f6ac408f4408   linux/arm64/v8   2723075                          
    100.    1   sha256:4d89eb6b628bc985ffcdcb939bf05a16194be6aed18c2848e1424d01ca0012fd   linux/arm64/v8   3193678                          
    101.    2   sha256:4818fb358d8f64c071010cb05fe59285888a36edbe1a45a5eed80576b08d312c   linux/arm64/v8       344                          
    102.  
    103. # Step6: Pushing to registry
    104. # Let us set up a local registry that we can push the encrypted image to. Note that by default, the docker/distribution registry version 2.7.1 and above supports encrypted OCI images out of the box.
    105. $ docker run -d -p 5000:5000 --restart=always --name registry registry:2.7.1
    106. Unable to find image 'registry:2.7.1' locally
    107. 2.7.1: Pulling from library/registry
    108. 486039affc0a: Pull complete 
    109. ba51a3b098e6: Pull complete 
    110. 8bb4c43d6c8e: Pull complete 
    111. 6f5f453e5f2d: Pull complete 
    112. 42bc10b72f42: Pull complete 
    113. Digest: sha256:7d081088e4bfd632a88e3f3bcd9e007ef44a796fddfe3261407a3f9f04abe1e7
    114. Status: Downloaded newer image for registry:2.7.1
    115. f65eff4accdc64eca42b5aa2c2c36e831c63172472800c835e031ec758a3ccbe
    116. $ docker ps |grep registry
    117. f65eff4accdc        registry:2.7.1                                 "/entrypoint.sh /etc…"   28 seconds ago      Up 27 seconds       0.0.0.0:5000->5000/tcp   registry
    118. $ $CTR images tag bash.enc:latest localhost:5000/bash.enc:latest
    119. localhost:5000/bash.enc:latest
    120. $ $CTR images list
    121. REF                            TYPE                                                      DIGEST                                                                  SIZE    PLATFORMS                                                                                LABELS 
    122. bash.enc:latest                application/vnd.oci.image.index.v1+json                   sha256:e1cea4c43a0d43bf3fdd6af66509668f6fe16e433085fcc7cf5b8a4ec48c80aa 5.7 MiB linux/386,linux/amd64,linux/arm/v6,linux/arm/v7,linux/arm64/v8,linux/ppc64le,linux/s390x -      
    123. docker.io/library/bash:latest  application/vnd.docker.distribution.manifest.list.v2+json sha256:8193ca23afc8e1069bd3d982733df79c68dbcfcd66b4da6b5d65da85987dae2f 5.7 MiB linux/386,linux/amd64,linux/arm/v6,linux/arm/v7,linux/arm64/v8,linux/ppc64le,linux/s390x -      
    124. localhost:5000/bash.enc:latest application/vnd.oci.image.index.v1+json                   sha256:e1cea4c43a0d43bf3fdd6af66509668f6fe16e433085fcc7cf5b8a4ec48c80aa 5.7 MiB linux/386,linux/amd64,linux/arm/v6,linux/arm/v7,linux/arm64/v8,linux/ppc64le,linux/s390x -      
    125. $ $CTR images push localhost:5000/bash.enc:latest
    126. [... truncated ...]
    127. elapsed: 0.4 s                                                                    total:  39.2 M (97.7 MiB/s)
    128. $ $CTR images rm localhost:5000/bash.enc:latest bash.enc:latest
    129. localhost:5000/bash.enc:latest
    130. bash.enc:latest
    131. $ $CTR images pull localhost:5000/bash.enc:latest
    132. localhost:5000/bash.enc:latest:                                                   resolved       |++++++++++++++++++++++++++++++++++++++| 
    133. index-sha256:e1cea4c43a0d43bf3fdd6af66509668f6fe16e433085fcc7cf5b8a4ec48c80aa:    done           |++++++++++++++++++++++++++++++++++++++| 
    134. manifest-sha256:043e2439dae1a9634be8b3a589c8a2e6048f0f1c0ca9d1516c53cd1ee8a8bdf8: done           |++++++++++++++++++++++++++++++++++++++| 
    135. layer-sha256:bdbc04a60398d706a06e5ae88a4cdbfef770c9fe1b12e31fcf5a50532dafad87:    done           |++++++++++++++++++++++++++++++++++++++| 
    136. layer-sha256:375abe9071e498e57f1cfcf349266948c3760bfb136e051c22a532d38a020e74:    done           |++++++++++++++++++++++++++++++++++++++| 
    137. layer-sha256:3298803dec193fdd561ea27d0b1469e3f0b5b796962f543f3bcc57cd12ebdf5d:    done           |++++++++++++++++++++++++++++++++++++++| 
    138. config-sha256:e155078e7721d0d550b869592482f0e2c9b9c40e6a541fb430ab46a278fde5cd:   exists         |++++++++++++++++++++++++++++++++++++++| 
    139. elapsed: 0.2 s                                                                    total:  3.0 Mi (15.1 MiB/s)                                      
    140. unpacking linux/amd64 sha256:e1cea4c43a0d43bf3fdd6af66509668f6fe16e433085fcc7cf5b8a4ec48c80aa...
    141. done
    142. $ $CTR images list
    143. REF                            TYPE                                                      DIGEST                                                                  SIZE    PLATFORMS                                                                                LABELS 
    144. docker.io/library/bash:latest  application/vnd.docker.distribution.manifest.list.v2+json sha256:8193ca23afc8e1069bd3d982733df79c68dbcfcd66b4da6b5d65da85987dae2f 5.7 MiB linux/386,linux/amd64,linux/arm/v6,linux/arm/v7,linux/arm64/v8,linux/ppc64le,linux/s390x -      
    145. localhost:5000/bash.enc:latest application/vnd.oci.image.index.v1+json                   sha256:e1cea4c43a0d43bf3fdd6af66509668f6fe16e433085fcc7cf5b8a4ec48c80aa 5.7 MiB linux/386,linux/amd64,linux/arm/v6,linux/arm/v7,linux/arm64/v8,linux/ppc64le,linux/s390x -     
    146.  
    147. # Step7: Decrypting the image
    148. $ $CTR images decrypt --key mykey.pem --platform linux/amd64 localhost:5000/bash.enc:latest localhost:5000/bash.dec:latest
    149. Decrypting localhost:5000/bash.enc:latest to localhost:5000/bash.dec:latest
    150. $ $CTR images list
    151. REF                            TYPE                                                      DIGEST                                                                  SIZE    PLATFORMS                                                                                LABELS 
    152. docker.io/library/bash:latest  application/vnd.docker.distribution.manifest.list.v2+json sha256:8193ca23afc8e1069bd3d982733df79c68dbcfcd66b4da6b5d65da85987dae2f 5.7 MiB linux/386,linux/amd64,linux/arm/v6,linux/arm/v7,linux/arm64/v8,linux/ppc64le,linux/s390x -      
    153. localhost:5000/bash.dec:latest application/vnd.oci.image.index.v1+json                   sha256:52dbac1ba1464568b2de8c9bc0d1b08d26786365e415c2c3ccb52a23191afc22 5.7 MiB linux/386,linux/amd64,linux/arm/v6,linux/arm/v7,linux/arm64/v8,linux/ppc64le,linux/s390x -      
    154. localhost:5000/bash.enc:latest application/vnd.oci.image.index.v1+json                   sha256:e1cea4c43a0d43bf3fdd6af66509668f6fe16e433085fcc7cf5b8a4ec48c80aa 5.7 MiB linux/386,linux/amd64,linux/arm/v6,linux/arm/v7,linux/arm64/v8,linux/ppc64le,linux/s390x -      
    155. # Notice that the 'DIGEST' of 'localhost:5000/bash.dec:latest' is different with the one of 'docker.io/library/bash:latest' 
    156. # Let's have a look at layerinfos of these images
    157. $ $CTR images layerinfo --platform linux/amd64 docker.io/library/bash:latest
    158.    #                                                                    DIGEST      PLATFORM      SIZE   ENCRYPTION   RECIPIENTS
    159.    0   sha256:c9b1b535fdd91a9855fb7f82348177e5f019329a58c53c47272962dd60f71fc9   linux/amd64   2802957                          
    160.    1   sha256:a3698fd3137d820dbb91b5f96e89af64b196dde4ab4c326dd1bd6291bbd771cf   linux/amd64   3186680                          
    161.    2   sha256:2538f4b330c99e67e96803569826ba62f7a3353d27f2a15bcf1e33d1814fa7ad   linux/amd64       340
    162. $ $CTR images layerinfo --platform linux/amd64 localhost:5000/bash.enc:latest
    163.    #                                                                    DIGEST      PLATFORM      SIZE   ENCRYPTION   RECIPIENTS
    164.    0   sha256:375abe9071e498e57f1cfcf349266948c3760bfb136e051c22a532d38a020e74   linux/amd64   2802957          jwe        [jwe]
    165.    1   sha256:3298803dec193fdd561ea27d0b1469e3f0b5b796962f543f3bcc57cd12ebdf5d   linux/amd64   3186680          jwe        [jwe]
    166.    2   sha256:bdbc04a60398d706a06e5ae88a4cdbfef770c9fe1b12e31fcf5a50532dafad87   linux/amd64       340          jwe        [jwe]
    167. $ $CTR images layerinfo --platform linux/amd64 localhost:5000/bash.dec:latest
    168.    #                                                                    DIGEST      PLATFORM      SIZE   ENCRYPTION   RECIPIENTS
    169.    0   sha256:c9b1b535fdd91a9855fb7f82348177e5f019329a58c53c47272962dd60f71fc9   linux/amd64   2802957                          
    170.    1   sha256:a3698fd3137d820dbb91b5f96e89af64b196dde4ab4c326dd1bd6291bbd771cf   linux/amd64   3186680                          
    171.    2   sha256:2538f4b330c99e67e96803569826ba62f7a3353d27f2a15bcf1e33d1814fa7ad   linux/amd64       340                          
    172. # layers of 'localhost:5000/bash.dec:latest' is same with 'docker.io/library/bash:latest' but the image 'DIGEST' differs 
    173.  
    174. # Step8: Running a container
    175. # We can now attempt to run the encrypted container image
    176. $ $CTR run --rm localhost:5000/bash.enc:latest test echo 'Hello World!'
    177. ctr: you are not authorized to use this image: missing private key needed for decryption
    178. # We notice that running the image failed. This is because we did not provide the keys for the encrypted image.We can pass the keys in with the --key flag.
    179. $ $CTR run --rm --key mykey.pem localhost:5000/bash.enc:latest test echo 'Hello World!'
    180. Hello World!

    Kubernetes Image Encryption Support

    目前社区计划对Kubernetes支持两种镜像加密模式：

    • Node Key Model：将密钥放在Kubernetes工作节点上，以节点为解密粒度(已实现)
    • Multitenant Key Model：多租户模型，支持解密粒度为集群和用户(还未实现，based on this KEP)

    本文主要探讨第一种模型，如图所示：

     

    原理很清晰：

    • 1、产生加密公钥和私钥，将密钥放在Kubernetes集群Worker节点指定路径下
    • 2、用公钥加密镜像并上传到镜像仓库
    • 3、用加密镜像在Kubernetes集群创建服务
    • 4、Worker节点Kubelet会调用container runtime拉取加密镜像，利用私钥对该镜像进行解密，最终成功运行解密后的镜像

    在了解大致流程后，我们进一步探讨一下Worker节点上container runtime对镜像加密的支持情况

    首先，Kubernetes使用CRI来对接container runtime，如图：

     

    而对于镜像加密特性，社区计划对container-runtime支持：containerd(+CRI-Containerd)以及cri-o，目前已经实现的有containerd(CRI-Containerd目前正在合并PR)以及cri-o。因此这里我们主要讨论containerd以及cri-o等计划支持镜像加密特性的container runtime实现

    这里给出cri结合cri-o，containerd以及docker的使用图示：

    通过上述图示，我们可以更加直观地看出Kubernetes对以上container runtime在使用上的细节

    Kubernetes Image Encryption Support-CRIO

    本章给出Kubernetes结合CRIO基于Node Key Model镜像加密模式的使用示例：

    1. # Step1: Install cri-o from source code
    2. # Install Runtime dependencies
    3. $ yum install -y \
    4.   containers-common \
    5.   device-mapper-devel \
    6.   git \
    7.   glib2-devel \
    8.   glibc-devel \
    9.   glibc-static \
    10.   go \
    11.   gpgme-devel \
    12.   libassuan-devel \
    13.   libgpg-error-devel \
    14.   libseccomp-devel \
    15.   libselinux-devel \
    16.   pkgconfig \
    17.   make \
    18.   runc
    19.  
    20. # Compile cri-o from source code
    21. $ git clone https://github.com/cri-o/cri-o go/src/github.com/cri-o/cri-o
    22. $ cd go/src/github.com/cri-o/cri-o && make
    23. mkdir -p "/root/go/src/github.com/cri-o/cri-o/_output/src/github.com/cri-o"
    24. ln -s "/root/go/src/github.com/cri-o/cri-o" "/root/go/src/github.com/cri-o/cri-o/_output/src/github.com/cri-o/cri-o"
    25. touch "/root/go/src/github.com/cri-o/cri-o/_output/.gopathok"
    26. GO111MODULE=on go build --mod=vendor -ldflags '-s -w -X main.gitCommit="186c23056ac25396e9ea9e49c5e5bfbe271b7751" -X main.buildInfo=1582724503' -tags "containers_image_ostree_stub  exclude_graphdriver_btrfs btrfs_noversion    seccomp selinux" -o bin/crio github.com/cri-o/cri-o/cmd/crio
    27. GO111MODULE=on go build --mod=vendor -ldflags '-s -w -X main.gitCommit="186c23056ac25396e9ea9e49c5e5bfbe271b7751" -X main.buildInfo=1582724526' -tags "containers_image_ostree_stub  exclude_graphdriver_btrfs btrfs_noversion    seccomp selinux" -o bin/crio-status github.com/cri-o/cri-o/cmd/crio-status
    28. make -C pinns
    29. make[1]: Entering directory `/root/go/src/github.com/cri-o/cri-o/pinns'
    30. cc -std=c99 -Os -Wall -Wextra -static   -c -o pinns.o pinns.c
    31. cc -o ../bin/pinns pinns.o -std=c99 -Os -Wall -Wextra -static 
    32. make[1]: Leaving directory `/root/go/src/github.com/cri-o/cri-o/pinns'
    33. ./bin/crio --config=""  config  > crio.conf
    34. (/root/go/src/github.com/cri-o/cri-o/build/bin/go-md2man -in docs/crio-status.8.md -out docs/crio-status.8.tmp && touch docs/crio-status.8.tmp && mv docs/crio-status.8.tmp docs/crio-status.8) || \
    35.         (/root/go/src/github.com/cri-o/cri-o/build/bin/go-md2man -in docs/crio-status.8.md -out docs/crio-status.8.tmp && touch docs/crio-status.8.tmp && mv docs/crio-status.8.tmp docs/crio-status.8)
    36. (/root/go/src/github.com/cri-o/cri-o/build/bin/go-md2man -in docs/crio.conf.5.md -out docs/crio.conf.5.tmp && touch docs/crio.conf.5.tmp && mv docs/crio.conf.5.tmp docs/crio.conf.5) || \
    37.         (/root/go/src/github.com/cri-o/cri-o/build/bin/go-md2man -in docs/crio.conf.5.md -out docs/crio.conf.5.tmp && touch docs/crio.conf.5.tmp && mv docs/crio.conf.5.tmp docs/crio.conf.5)
    38. (/root/go/src/github.com/cri-o/cri-o/build/bin/go-md2man -in docs/crio.8.md -out docs/crio.8.tmp && touch docs/crio.8.tmp && mv docs/crio.8.tmp docs/crio.8) || \
    39.         (/root/go/src/github.com/cri-o/cri-o/build/bin/go-md2man -in docs/crio.8.md -out docs/crio.8.tmp && touch docs/crio.8.tmp && mv docs/crio.8.tmp docs/crio.8)
    40. $ make install  
    41. install  -D -m 755 bin/crio /usr/local/bin/crio
    42. install  -D -m 755 bin/crio-status /usr/local/bin/crio-status
    43. install  -D -m 755 bin/pinns /usr/local/bin/pinns
    44. install  -d -m 755 /usr/local/share/man/man5
    45. install  -d -m 755 /usr/local/share/man/man8
    46. install  -m 644 docs/crio.conf.5 -t /usr/local/share/man/man5
    47. install  -m 644 docs/crio-status.8 docs/crio.8 -t /usr/local/share/man/man8
    48. install  -d -m 755 /usr/local/share/bash-completion/completions
    49. install  -d -m 755 /usr/local/share/fish/completions
    50. install  -d -m 755 /usr/local/share/zsh/site-functions
    51. install  -D -m 644 -t /usr/local/share/bash-completion/completions completions/bash/crio
    52. install  -D -m 644 -t /usr/local/share/fish/completions completions/fish/crio.fish
    53. install  -D -m 644 -t /usr/local/share/zsh/site-functions  completions/zsh/_crio
    54. install  -D -m 644 -t /usr/local/share/bash-completion/completions completions/bash/crio-status
    55. install  -D -m 644 -t /usr/local/share/fish/completions completions/fish/crio-status.fish
    56. install  -D -m 644 -t /usr/local/share/zsh/site-functions  completions/zsh/_crio-status
    57.  
    58. # Download conmon
    59. $ git clone https://github.com/containers/conmon ~/go/src/github.com/containers/conmon
    60. $ cd ~/go/src/github.com/containers/conmon
    61. $ make  
    62. mkdir -p bin
    63. cc -std=c99 -Os -Wall -Wextra -Werror -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include   -DVERSION=\"2.0.11-dev\" -DGIT_COMMIT=\""86aa80b908d7532ab9b6edd6f4f27ba4bf6ba17b"\"   -D USE_JOURNALD=0 -o src/conmon.o -c src/conmon.c
    64. cc -std=c99 -Os -Wall -Wextra -Werror -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include   -DVERSION=\"2.0.11-dev\" -DGIT_COMMIT=\""86aa80b908d7532ab9b6edd6f4f27ba4bf6ba17b"\"   -D USE_JOURNALD=0 -o src/cmsg.o -c src/cmsg.c
    65. cc -std=c99 -Os -Wall -Wextra -Werror -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include   -DVERSION=\"2.0.11-dev\" -DGIT_COMMIT=\""86aa80b908d7532ab9b6edd6f4f27ba4bf6ba17b"\"   -D USE_JOURNALD=0 -o src/ctr_logging.o -c src/ctr_logging.c
    66. cc -std=c99 -Os -Wall -Wextra -Werror -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include   -DVERSION=\"2.0.11-dev\" -DGIT_COMMIT=\""86aa80b908d7532ab9b6edd6f4f27ba4bf6ba17b"\"   -D USE_JOURNALD=0 -o src/utils.o -c src/utils.c
    67. cc  -o bin/conmon src/conmon.o src/cmsg.o src/ctr_logging.o src/utils.o -lglib-2.0   -lsystemd 
    68. $ make install
    69. install  -D -m 755 bin/conmon /usr/local/bin/conmon
    70.  
    71. # Setup CNI networking
    72. # Set CNI network configurations
    73. $ cd /root/go/src/github.com/cri-o/cri-o && mkdir -p /etc/cni/net.d
    74. $ cp contrib/cni/*.conf /etc/cni/net.d/
    75. $ ls /etc/cni/net.d/
    76. 10-crio-bridge.conf  99-loopback.conf
    77. # Install the CNI plugins
    78. $ git clone https://github.com/containernetworking/plugins ~/go/src/github.com/containernetworking/plugins
    79. $ cd ~/go/src/github.com/containernetworking/plugins && git checkout v0.8.5
    80. $ ./build_linux.sh
    81. Building plugins 
    82.   bandwidth
    83.   firewall
    84.   flannel
    85.   portmap
    86.   sbr
    87.   tuning
    88.   bridge
    89.   host-device
    90.   ipvlan
    91.   loopback
    92.   macvlan
    93.   ptp
    94.   vlan
    95.   dhcp
    96.   host-local
    97.   static
    98. $ mkdir -p /opt/cni/bin
    99. $ cp bin/* /opt/cni/bin/
    100.  
    101. # Generating CRI-O configuration
    102. $ cd ~/go/src/github.com/cri-o/cri-o/ && make install.config
    103. install  -d /usr/local/share/containers/oci/hooks.d
    104. install  -D -m 644 crio.conf /etc/crio/crio.conf
    105. install  -D -m 644 crio-umount.conf /usr/local/share/oci-umount/oci-umount.d/crio-umount.conf
    106. install  -D -m 644 crictl.yaml /etc
    107. # Path to OCI hooks directories for automatically executed hooks.
    108. hooks_dir = [
    109.         "/usr/local/share/containers/oci/hooks.d",
    110. ]
    111.  
    112. # Validate registries in registries.conf
    113. # Edit /etc/containers/registries.conf and verify that the registries option has valid values in it.
    114. $ cat /etc/containers/registries.conf
    115. [... truncated ...]
    116. [registries.search]
    117. registries = ['registry.access.redhat.com', 'docker.io', 'registry.fedoraproject.org', 'quay.io', 'registry.centos.org']
    118.  
    119. [registries.insecure]
    120. registries = []
    121.  
    122. [registries.block]
    123. registries = []
    124.  
    125. # Recommended - Use systemd cgroups.
    126. # By default, CRI-O uses cgroupfs as a cgroup manager. However, we recommend using systemd as a cgroup manager. You can change your cgroup manager in crio.conf:
    127. cgroup_manager = "systemd"
    128.  
    129. # Optional - Modify verbosity of logs in /etc/crio/crio.conf
    130. # Users can modify the log_level field in /etc/crio/crio.conf to change the verbosity of the logs. Options are fatal, panic, error (default), warn, info, and debug.
    131. log_level = "info"
    132.  
    133. # Starting CRI-O
    134. $ make install.systemd
    135. install  -D -m 644 contrib/systemd/crio.service /usr/local/lib/systemd/system/crio.service
    136. ln -sf crio.service /usr/local/lib/systemd/system/cri-o.service
    137. install  -D -m 644 contrib/systemd/crio-shutdown.service /usr/local/lib/systemd/system/crio-shutdown.service
    138. install  -D -m 644 contrib/systemd/crio-wipe.service /usr/local/lib/systemd/system/crio-wipe.service
    139. $ systemctl daemon-reload
    140. $ systemctl enable crio
    141. Created symlink from /etc/systemd/system/multi-user.target.wants/crio.service to /usr/local/lib/systemd/system/crio.service.
    142. # enable crio proxy(pulling image)
    143. $ cat /usr/local/lib/systemd/system/crio.service
    144. [... truncated ...]
    145. [Service]
    146. Environment="GOTRACEBACK=crash" "HTTP_PROXY=x.x.x.x" "HTTPS_PROXY=x.x.x.x" "NO_PROXY=localhost,127.0.0.1"
    147. $ systemctl start crio
    148. $ systemctl status crio
    149. * crio.service - Container Runtime Interface for OCI (CRI-O)
    150.    Loaded: loaded (/usr/local/lib/systemd/system/crio.service; enabled; vendor preset: disabled)
    151.    Active: active (running) since Wed 2020-02-26 22:27:38 CST; 1min 11s ago
    152.      Docs: https://github.com/cri-o/cri-o
    153.  Main PID: 16267 (crio)
    154.    CGroup: /system.slice/crio.service
    155.            `-16267 /usr/local/bin/crio
    156.  
    157. Feb 26 22:27:37 vm-xxx-centos systemd[1]: Starting Container Runtime Interface for OCI (CRI-O)...
    158. Feb 26 22:27:37 vm-xxx-centos crio[16267]: time="2020-02-26 22:27:37.855228868+08:00" level=warning msg="Not using nat...o fix"
    159. Feb 26 22:27:37 vm-xxx-centos crio[16267]: time="2020-02-26 22:27:37.855645605+08:00" level=info msg="Found CNI networ....conf"
    160. Feb 26 22:27:37 vm-xxx-centos crio[16267]: time="2020-02-26 22:27:37.855703783+08:00" level=info msg="Found CNI networ....conf"
    161. Feb 26 22:27:38 vm-xxx-centos crio[16267]: W0226 22:27:38.114732   16267 hostport_manager.go:69] The binary conntrack ...eanup.
    162. Feb 26 22:27:38 vm-xxx-centos crio[16267]: time="2020-02-26 22:27:38.114771405+08:00" level=info msg="no seccomp profi...fault"
    163. Feb 26 22:27:38 vm-xxx-centos systemd[1]: Started Container Runtime Interface for OCI (CRI-O).  
    164.  
    165. # Step2: Install crictl
    166. $ VERSION="v1.17.0"
    167. $ wget https://github.com/kubernetes-sigs/cri-tools/releases/download/$VERSION/crictl-$VERSION-linux-amd64.tar.gz
    168. $ tar zxvf crictl-$VERSION-linux-amd64.tar.gz -C /usr/local/bin
    169. crictl
    170. $ rm -f crictl-$VERSION-linux-amd64.tar.gz
    171. # To ensure that we’ve setup CRI-O correctly, we will install crictl to verify our setup.
    172. $ crictl -r unix:///run/crio/crio.sock info 
    173. {
    174.   "status": {
    175.     "conditions": [
    176.       {
    177.         "type": "RuntimeReady",
    178.         "status": true,
    179.         "reason": "",
    180.         "message": ""
    181.       },
    182.       {
    183.         "type": "NetworkReady",
    184.         "status": true,
    185.         "reason": "",
    186.         "message": ""
    187.       }
    188.     ]
    189.   }
    190. }
    191.  
    192. # Step3: Setting up a single node k8s cluster
    193. $ wget https://github.com/kubernetes/kubernetes/archive/v1.17.1.tar.gz
    194. $ tar -xzf v1.17.1.tar.gz && cd kubernetes-1.17.1
    195. $ hack/install-etcd.sh   
    196. Downloading https://github.com/coreos/etcd/releases/download/v3.4.3/etcd-v3.4.3-linux-amd64.tar.gz succeed
    197. etcd v3.4.3 installed. To use:
    198. export PATH="/root/go/src/github.com/kubernetes-1.17.1/third_party/etcd:${PATH}"
    199. $ export PATH="/root/go/src/github.com/kubernetes-1.17.1/third_party/etcd:${PATH}"
    200. $ CGROUP_DRIVER=systemd \
    201. CONTAINER_RUNTIME=remote \
    202. CONTAINER_RUNTIME_ENDPOINT='unix:///var/run/crio/crio.sock' \
    203. ./hack/local-up-cluster.sh
    204. [... truncated ...]
    205. To start using your cluster, you can open up another terminal/tab and run:
    206.  
    207.   export KUBECONFIG=/var/run/kubernetes/admin.kubeconfig
    208.   cluster/kubectl.sh
    209.  
    210. Alternatively, you can write to the default kubeconfig:
    211.  
    212.   export KUBERNETES_PROVIDER=local
    213.  
    214.   cluster/kubectl.sh config set-cluster local --server=https://localhost:6443 --certificate-authority=/var/run/kubernetes/server-ca.crt
    215.   cluster/kubectl.sh config set-credentials myself --client-key=/var/run/kubernetes/client-admin.key --client-certificate=/var/run/kubernetes/client-admin.crt
    216.   cluster/kubectl.sh config set-context local --cluster=local --user=myself
    217.   cluster/kubectl.sh config use-context local
    218.   cluster/kubectl.sh
    219. $ cluster/kubectl.sh get nodes
    220. NAME        STATUS   ROLES    AGE   VERSION
    221. 127.0.0.1   Ready    <none>   21m   v1.17.1
    222.  
    223. # Step4: Generating encrypted image with skopeo
    224. # Install skopeo
    225. $ yum install libgpgme-devel device-mapper-devel libbtrfs-devel glib2-devel libassuan-devel
    226. $ git clone https://github.com/containers/skopeo $GOPATH/src/github.com/containers/skopeo
    227. $ cd $GOPATH/src/github.com/containers/skopeo && make binary-local
    228. $ make install
    229. $ mkdir /etc/containers
    230. $ cp default-policy.json /etc/containers/policy.json
    231. # Pull image
    232. $ skopeo copy docker://docker.io/library/nginx:1.15 oci:local_nginx:1.15
    233. Getting image source signatures
    234. Copying blob 743f2d6c1f65 done  
    235. Copying blob 6bfc4ec4420a done  
    236. Copying blob 688a776db95f done  
    237. Copying config 0fb15759be done  
    238. Writing manifest to image destination
    239. Storing signatures
    240. # Encrypt image
    241. $ skopeo  copy --encryption-key jwe:./mypubkey.pem oci:local_nginx:1.15 oci:nginx_encrypted:1.15
    242. Getting image source signatures
    243. Copying blob 743f2d6c1f65 done  
    244. Copying blob 6bfc4ec4420a done  
    245. Copying blob 688a776db95f done  
    246. Copying config 0fb15759be done  
    247. Writing manifest to image destination
    248. Storing signatures
    249. # Push image
    250. $ skopeo  copy --dest-tls-verify=false oci:nginx_encrypted:1.15 docker://x.x.x.x/nginx_encrypted:1.15  
    251. Getting image source signatures
    252. Copying blob cd2a4504255e done  
    253. Copying blob 8df8b398a84d done  
    254. Copying blob 663a05dac0ac done  
    255. Copying config 0fb15759be done  
    256. Writing manifest to image destination
    257. Storing signatures
    258.  
    259. # Step5: Getting and running an encrypted image
    260. $ cat <<EOF > enc-dply.yaml
    261. apiVersion: apps/v1
    262. kind: Deployment
    263. metadata:
    264.   name: enc-nginx
    265.   labels:
    266.     app: nginx
    267. spec:
    268.   replicas: 3
    269.   selector:
    270.     matchLabels:
    271.       app: nginx
    272.   template:
    273.     metadata:
    274.       labels:
    275.         app: nginx
    276.     spec:
    277.       containers:
    278.       - name: nginx
    279.         image: x.x.x.x/nginx_encrypted:1.15
    280.         ports:
    281.         - containerPort: 80
    282. EOF
    283. $ cluster/kubectl.sh create -f enc-dply.yaml
    284. deployment.apps/enc-nginx created
    285. $ cluster/kubectl.sh get pods
    286. NAME                         READY   STATUS             RESTARTS   AGE
    287. enc-nginx-7fb4578896-9qhtr   0/1     ImagePullBackOff   0          12s
    288. enc-nginx-7fb4578896-cdrdq   0/1     ErrImagePull       0          12s
    289. enc-nginx-7fb4578896-j94rj   0/1     ErrImagePull       0          12s
    290. # We notice that our pods failed to run due to failure in the image pull process. Let’s describe the pod to find out more.
    291. $ cluster/kubectl.sh describe pods/enc-nginx-7fb4578896-9qhtr
    292. [... truncated ...]
    293. Events:
    294.   Type     Reason     Age                From                Message
    295.   ----     ------     ----               ----                -------
    296.   Normal   Scheduled  <unknown>          default-scheduler   Successfully assigned default/enc-nginx-7fb4578896-9qhtr to 127.0.0.1
    297.   Normal   Pulling    25s (x2 over 36s)  kubelet, 127.0.0.1  Pulling image "x.x.x.x/nginx_encrypted:1.15"
    298.   Warning  Failed     25s (x2 over 36s)  kubelet, 127.0.0.1  Failed to pull image "x.x.x.x/nginx_encrypted:1.15": rpc error: code = Unknown desc = Error decrypting layer sha256:cd2a4504255eaf69fb1ee6c5961625854cd69597a1c5722e86062fb4ab688cf8: missing private key needed for decryption
    299.   Warning  Failed     25s (x2 over 36s)  kubelet, 127.0.0.1  Error: ErrImagePull
    300.   Normal   BackOff    13s (x2 over 36s)  kubelet, 127.0.0.1  Back-off pulling image "x.x.x.x/nginx_encrypted:1.15"
    301.   Warning  Failed     13s (x2 over 36s)  kubelet, 127.0.0.1  Error: ImagePullBackOff
    302. # Configuring decryption keys for CRI-O
    303. $ mkdir -p /etc/crio/keys
    304. $ cat << EOF > /etc/crio/keys/mykey.pem
    305. -----BEGIN RSA PRIVATE KEY-----
    306. MIIEpAIBAAKCAQEA3B07uvDP+NxYXYQxjknPml1zSacijqoj1D79VzSUjjMM6th/
    307. doKJbeMTybehF9PDly6TleluZWZiXMtqagmeEI/sUKaaq0yrMrNrzfIfJYCD0hJx
    308. 8tBFZKwUVKaYQjB4Bo6Ij0V04dcP8m6NAfXNG0/MB56zj7WhMndOPP/tuIGmj1jV
    309. jBopq/Pm8KUrzDVCmD+kQLT7hZTi0lapG7A/tE7bc2pGypP2hLR0i9/ckieXh4IL
    310. YKdaNda4THjnA3bHC7/ELOWPinrmQlaxY1mW8hkLCHCQ/MG/SJKkVTY2fX62BscY
    311. KGQPOwh9jNWIS8MAbNJafYBxVueje6Xe/e+OdwIDAQABAoIBAQDXOH4+u1eerVR5
    312. m9gYmHM1LEqdqZ5QgGuoDC8KJY9buu7WcfmvltNpbq7afYI2GgkUuaX03tniq8lh
    313. kkPqipzS9ObLtRtmgwCiAm1WYXey44YA0ag5EwvG87qtSnd1wI6bWqKL9A3lBLPD
    314. B/U4BW8XVV7Z1IMd8So8fgsx+cwmqk2+CqMZsq2m7FBjrCt7fu0QMaOXvuUWka5D
    315. K8F+yvAbD5UMXOqN9fjMuPzbtiOFXvmWtpC3L6wxZjmAPuiR9fNzwM4jCXSasMU8
    316. PNy3x0Esn/ic7IP+1v1PEiXImLui4WPF74v8i6HB6FgxCBht6rCYyl8kYh68gJS6
    317. HTmgf9+BAoGBAPk+ft+R4T/iGDZ0y1YhRNN5zxqtOEcLf1+5/s6W6b+FTRFrxYLL
    318. A8ZMkn0Se8DfVjPald1gMSikE8vj8YJSHAMlstPiHiYQKsBMGbjksZLjBpSapkTy
    319. iMjpk+SmVN3sZRU3jfrcMoPfAQx0hBrCUJdhZFV/aYLiuJ7GUub34gKzAoGBAOIU
    320. mu5aht3MDjiR2mnNzEj2TY7YED7HNWOQJ0FzqLfl4jZTowMbnz8aFUWxkJmC3i1y
    321. 0Y8rWbxV7CwcXc9fOR8jRNx2Jn6eaA4A9+a02D1hNHJ6bQivy6yloQXCs4/wardY
    322. EBHvepygs+v507khX146bl8PmUCIYpjVFaeOrRctAoGAehhLPmnP1eODyOld0ktp
    323. 086PzZmdP/A57ULHt5vl1ZQPNMF+d5vLtZA9ElfDl6/QIoapc1BzxFzb9b0ryZM/
    324. das59uGFs0+oIZsl3pTpB/N+fb1kRdIpf4IsmI2CdVQgEEyumHzVohPUB63sKM+X
    325. exCSfe90WFGH7v9oDQzRAlECgYBocSRx4JhVdqNLNvYz0sMBIegKiX5XwifD6yB3
    326. eDsFWcn7VwADu4sB18bj/3fRs0d4r4ZoIZq/CuKkLiaYWmFFJUH2pw55iCyB66ia
    327. iAktse5MxIoCbVQmWg3dX2kcofBq6t/hqUR3fzYfWbaZ2/T2zv+WItqlmVwTRr1O
    328. PvdvsQKBgQDA0MQnQ7iV6Uvdm2AFHXlnwtXGelG0EsaNabF31LosHujMf92kUzaC
    329. 12NiuE0gEfuPemEw+MFQ/nfBiEDv23NgRz6frZvcvUYWr/xFs8AwPkV5pBFNwoa2
    330. 4lYPbMB/1RWf4zDKvDBKEpEGr3pCKBjrPszi0skn2DTCcpiqjH3XGg==
    331. -----END RSA PRIVATE KEY-----
    332. EOF
    333. $ cluster/kubectl.sh delete -f enc-dply.yaml
    334. deployment.apps "enc-nginx" deleted
    335. $ cluster/kubectl.sh create -f enc-dply.yaml
    336. deployment.apps/enc-nginx created
    337. $ cluster/kubectl.sh get pods -o wide
    338. NAME                         READY   STATUS    RESTARTS   AGE   IP            NODE        NOMINATED NODE   READINESS GATES
    339. enc-nginx-7fb4578896-8grjd   1/1     Running   0          61s   10.88.0.102   127.0.0.1   <none>           <none>
    340. enc-nginx-7fb4578896-jjhz4   1/1     Running   0          61s   10.88.0.100   127.0.0.1   <none>           <none>
    341. enc-nginx-7fb4578896-jv9n5   1/1     Running   0          61s   10.88.0.101   127.0.0.1   <none>           <none>
    342. # test with curl
    343. $ curl -v 10.88.0.92
    344. < HTTP/1.1 200 OK
    345. < Server: nginx/1.15.12
    346. < Date: Thu, 27 Feb 2020 09:47:44 GMT
    347. < Content-Type: text/html
    348. < Content-Length: 612
    349. < Last-Modified: Tue, 16 Apr 2019 13:08:19 GMT
    350. < Connection: keep-alive
    351. < ETag: "5cb5d3c3-264"
    352. < Accept-Ranges: bytes
    353. < 
    354. <!DOCTYPE html>
    355. <html>
    356. <head>
    357. <title>Welcome to nginx!</title>
    358. <style>
    359.     body {
    360.         width: 35em;
    361.         margin: 0 auto;
    362.         font-family: Tahoma, Verdana, Arial, sans-serif;
    363.     }
    364. </style>
    365. </head>
    366. <body>
    367. <h1>Welcome to nginx!</h1>
    368. <p>If you see this page, the nginx web server is successfully installed and
    369. working. Further configuration is required.</p>
    370.  
    371. <p>For online documentation and support please refer to
    372. <a href="http://nginx.org/">nginx.org</a>.<br/>
    373. Commercial support is available at
    374. <a href="http://nginx.com/">nginx.com</a>.</p>
    375.  
    376. <p><em>Thank you for using nginx.</em></p>
    377. </body>
    378. </html>

    The main integration points for Encrypted Container Images

    镜像加密工具

    对于镜像加密工具，社区计划支持：docker CLI(To integrate into docker CLI, we are currently waiting on moby/moby#38043. Tracking with moby/buildkit#714), containerd imgcrypt, skopeo以及buildah，目前已经实现的有containerd imgcrypt和skopeo

    镜像仓库

    Docker Distribution >= v2.7.1 支持镜像加密OCI格式

    Container Runtime

    于Container Runtime，社区计划支持：containerd(+CRI-Containerd)以及cri-o(暂不支持Docker)，目前已经实现的有containerd(CRI-Containerd目前正在合并PR)以及cri-o

    Kubernetes

    目前社区计划对Kubernetes支持两种镜像加密模式：

    • Node Key Model：将密钥放在Kubernetes工作节点上，以节点为解密粒度(CRIO已经实现，CRI-Containerd待合并PR)
    • Multitenant Key Model：多租户模型，支持解密粒度为集群和用户(还未实现，based on this KEP)

    另外，对于Node Key Model模式，测试加密流程可行，并对性能评估如下：

    • 1、目前社区对镜像加密这块并没有性能测试数据
    • 2、利用对称加密算法(i.e. AES)加密镜像Layer，加密后数据长度不变——对镜像pull影响不大&解密速度快，耗费时间少
    • 3、Memory would not be that bad since we are using a stream cipher

    补充：目前社区对于Docker的支持计划参考如下(我的思考)：应该是等到Docker底层代码切换到Containerd(moby/moby#38043)，然后修改docker-shim支持镜像加密，所以目前是阻塞的状态

    Conclusion

    ​在容器安全涉及的内容中，目前还不十分成熟的部分是镜像加密，镜像加密在数据保密性要求较强的领域中会显得十分重要，例如：金融、银行以及国企等

    本文首先介绍了镜像加密涉及的OCI规范，在此基础上对镜像加密原理以及流程进行了说明，并利用containerd imgcrypt工具对加密流程进行了实操。而Kubernetes对镜像加密特性支持Node Key Model以及Multitenant Key Model这两种使用模式，本文详细讲解了Kubernetes基于Node Key Model使用模式的原理&流程，并结合CRIO演示了镜像加密Kubernetes-Native的用法

    虽然目前基于社区的工作可以在Kubernetes上使用镜像加密特性，但是该特性的支持还并不完善，比如：对于加密工具，还需要支持docker CLI(To integrate into docker CLI, we are currently waiting on moby/moby#38043. Tracking with moby/buildkit#714)；对于容器运行时，还需要支持CRI-Containerd以及Docker等；对于云原生生态系统，还需要更多地与镜像相关的项目(such as Notary, Clair and so on)进行合作；对于Kubernetes，还需要支持更多Key Models

    Refs

  • 相关阅读:
    【数据结构】----枚举
    I420转RGB24，YUY2转RGB24，RGB24垂直翻转，RGB24水平翻转
    【问题思考总结】数据链路层和传输层的功能是否重复冗余？（差错控制与流量控制）
    QCC51XX---GaiaFramework_Init分析
    Js基础:JS中怎么将数据转为布尔值
    工程机械比例阀电流采集方案
    Smile or Scowl? Looking at Infographic DesignThrough the Affective Lens
    设计模式之单例和原型
    Vue3：用vite创建Vue3项目
    美业连锁门店收银系统源码-如何查看收款门店对应的加盟商？
  • 原文地址：https://blog.csdn.net/ATYtian/article/details/125475345