DevOps --- Pipeline和Yaml文件

什么是Pipleine

• 在DevOps中pipeline可以按顺序在一台虚拟机上实现一组特定的操作，包括但不限于

• 执行git命令，如拉取代码，push代码等
• 执行任意程序
• 执行python或bash脚本
• …

• 基于以上操作，pipeline通常可以完成以下场景

• 持续集成：拉取代码，运行代码质量检查工具，运行集成测试，部署代码等
• 使用第三方控件对代码进行安全扫描(DAST, Image,SAST等)
• 定时运行E2E测试
• …

什么是Yaml文件

• 和Json，XML类似，YAML（YAML Ain’t Markup Language）是一种数据序列化格式。经常被用于配置文件和数据交换格式，特别适用于各种编程语言之间的数据传递.
• 在微软的DevOps中，yaml通常用来定义一个pipeline

Yaml文件的构成

• 一个pipeline 由多个stages组成
• 一个stage由多个jobs组成

• 每个Job可以在一个agent上执行（一个Job也可以没有agent）
• an agent could be an VM or sometimes an VM can have multiple agents

• 一个job由多个steps组成

• steps可以是task或script，是pipeline的最小构建单元

• task是一些Azure DevOps预先设定好的功能，可以直接使用. 常用的task包括,

• PythonScript@0
• PowerShell@2
• Docker@1
• CopyFiles@2
• PublishTestResults@2
• …
• 可以在ADO中搜索
  

• Trigger

• This states what changes trigger the pipeline. For the example in the below code, changes in the main branch alone trigger the pipeline run and not from the feature branches.

• Pool

• When your build or deployment runs, the system begins one or more jobs. An agent is a computing infrastructure with installed agent software that runs one job at a time. For example, your job could run on a Microsoft-hosted Ubuntu agent. The below pipeline yaml uses “ubuntu-latest”

• Variables

• Variables are a way to get key bits of data into various parts of the pipeline.
• The most common use of variables is to define a value that you can then use in your pipeline.
• All variables are strings and are mutable. The value of a variable can change from run to run or job to job of your pipeline.

Example

trigger:
  branches:
    include:
    - main
    exclude:
    - feature_branches

pool:
  vmImage: 'ubuntu-latest'

variables:
  - name: tag
    value: '$(Build.BuildNumber)'
  - name: ImageName
    value: "demo Image"
  - name: python.version
    value: '3.8'

schedules:
 - cron: "0 18 1 * *"
   always: true
   displayName: Monthly, 2nd, 2:00 am (utc+8)
   branches:
     include:
     - master

stages:
  - stage: Lint
    displayName: Format and lint code
    jobs:
      - job: Linting
        steps:
        - script: |
            python3 -m pip install black
            python3 -m pip install pylint
          displayName: "Install dependencies"  

        - script: |
            #app is the folder in which the application code resides
            python3 -m black ./app
          displayName: "Apply black code formatting"

        - script: |
            python3 -m pylint ./app --recursive=true --exit-zero
          displayName: "Static code analysis with pylint"
  
  - stage: Test
    displayName: Unit test
    jobs:
      - job: Test
        steps:
        - script: |
            python3 -m pip install -r requirements.txt
            python3 -m pip install pytest-azurepipelines
          displayName: "Install dependencies"
        
        - script: |
            python3 -m pytest -v -s --junitxml=unittest/junit.xml --cov=. --cov-report=term-missing --cov-report=xml
  
  - stage: Build
    displayName: 'Build and Push Docker Image'
    jobs:
      - job: BuildAndPush
        displayName: 'Build and Push Docker Image'
        steps:
       - task: Docker@1
         displayName: 'Build an image'
        inputs:
        containerregistrytype: 'Azure Container Registry'
        azureSubscriptionEndpoint: 'Service connection name'
        azureContainerRegistry: '<>'
        command: 'Build an Image'
        dockerFile: '$(System.DefaultWorkingDirectory)/Dockerfile'
        tags: |
          latest
          $(Build.BuildId)
        imageName: '$(ImageName):$(tag)'

       - task: Docker@1
         displayName: 'Push image to ACR to TEST'
         inputs:
           containerregistrytype: 'Azure Container Registry'
           azureSubscriptionEndpoint: '<>'
           azureContainerRegistry: 'container registary name-DEV.azurecr.io'
           command: 'Push an image'
           imageName: '$(ImageName):$(tag)'
      
       - task: Docker@1
         displayName: 'Build an image'
         inputs:
           containerregistrytype: 'Azure Container Registry'
           azureSubscriptionEndpoint: 'Service connection name'
           azureContainerRegistry: 'container registary name-PROD.azurecr.io'
           command: 'Build an Image'
           dockerFile: '$(System.DefaultWorkingDirectory)/Dockerfile'
           tags: |
               latest
               $(Build.BuildId)
           imageName: '$(ImageName):$(tag)'
      
       - task: Docker@1
         displayName: 'Push image to ACR to PROD'
         inputs:
           containerregistrytype: 'Azure Container Registry'
           azureSubscriptionEndpoint: 'Service connection name'
           azureContainerRegistry: 'container registary name-PROD.azurecr.io'
           command: 'Push an image'
           imageName: '$(ImageName):$(tag)'      
           
       - task: CopyFiles@2
         inputs:
           SourceFolder: 'kubernetes'
           Contents: '*.yaml'
           TargetFolder: '$(Build.ArtifactStagingDirectory)'
      
       - task: PublishBuildArtifacts@1
         inputs:
           PathtoPublish: '$(Build.ArtifactStagingDirectory)'
           ArtifactName: 'manifest'
           publishLocation: 'Container'