Nightly batch run on Azure DevOps Pipeline

My choice of solution is

• Create a console app that does the job.
• Set up a pipeline that runs nightly to execute the console app.

CI Build

First, build a console app and publish it as artifact.

name: 1.0$(Rev:.r)

trigger:
  branches:
    include:
    - master
    - hotfix/*
    - release/*
    - feature/*
    - bugfix/*
    - backlog/*
    - task/*

variables:
  major: 1
  minor: 0
  patch: $[counter(variables['minor'], 0)]
  version: $(major).$(minor).$(patch)
  dotnetCliVersion: 3.1.101
  buildConfiguration: Release
  buildPlatform: x64
  buildProjects: "**/*.csproj"
  cliProject: '**/*Clis.csproj'

jobs:
  - job: build_solution
    pool:
      vmImage: windows-2019
    steps:
      - checkout: self
        clean: true
        
      - task: UseDotNet@2        
        inputs:
          packageType: 'sdk'
          version: $(dotnetCliVersion)
          displayName: dotnet install

      # Restore packages
      - task: DotNetCoreCLI@2
        displayName: Restore Packages
        inputs:
          command: restore
          feedsToUse: select
          vstsFeed: 'innovation-release'
          includeNuGetOrg: false
          projects: $(buildProjects)
          noCache: false

      # Build          
      - task: DotNetCoreCLI@2
        displayName: Build Projects
        inputs:
          projects: $(buildProjects)
          packDirectory: '$(Build.ArtifactStagingDirectory)'
          arguments: '-c $(buildConfiguration) /p:Platform=x64 /p:Version=$(Build.BuildNumber) --no-restore'      

      # Publish the cli application
      - task: DotNetCoreCLI@2
        displayName: Publish Console Project
        inputs:
          command: publish
          publishWebProjects: false
          zipAfterPublish: false
          projects: $(cliProject)
          arguments: --no-build --no-restore -c Release /p:Platform=x64 /p:Version=$(Build.BuildNumber) -o "$(Build.ArtifactStagingDirectory)\cli\"

      - publish: $(Build.ArtifactStagingDirectory)\cli
        displayName: Publish Artifact
        artifact: cli

      - pwsh: |
          Write-Host "##vso[build.addbuildtag]$(version)"
        displayName: add build tag

A few things to note

• build.artifactStagingDirectory: the directory where artifact is staged

Nightly Run Pipeline

Now, I have a build artifact that contains the console application. Create another pipeline that will download the artifact and run the executable.

name: $(Build.DefinitionName)-$(Date:yyyyMMdd)$(Rev:.r)
schedules:
  - cron: "0 0 * * *"
    displayName: Nightly run
    always: true
    branches:
      include:
        - master
trigger: none
parameters:  
  - name: build_tag
    displayName: Build Tag
    type: string
    default: latest
variables:
  - name: major
    value: 1
  - name: minor
    value: 0
  - name: patch
    value: $[counter(variables['minor'], 0)]
  - name: version
    value: $(major).$(minor).$(patch)
resources:
  repositories:
    - repository: templates
      type: git
      name: /yaml-templates
      ref: refs/heads/master      
  pipelines:
    - pipeline: build
      source: ci

jobs:
  - job: run_cli
    pool:
      vmImage: windows-2019
    timeoutInMinutes: 90
    steps:
      - download: none
      - task: DownloadPipelineArtifact@2
        displayName: download cli from '$(resources.pipeline.build.pipelineName)' with tag '$'
        name: download_artifact_console
        inputs:
          source: specific
          project: $(System.TeamProjectId)
          pipeline: $(resources.pipeline.build.pipelineID)
          artifactName: cli
          path: $(Pipeline.Workspace)/cli
          runVersion: latest
          allowPartiallySucceededBuilds: false
          allowFailedBuilds: false
      # Execute cli app
      - pwsh: ./Clis.exe --cmd create-domainpage
        workingDirectory: $(Pipeline.Workspace)/cli
        displayName: Executing script
        env:
          wiki-token: $(wiki-token)

timeoutInMinutes is very handy. Be default, if a job runs longer than 60 minutes, it fails. One of my nightly job runs longer, so I set it to 90 minutes.

Lastly, we need to the access token for the cli app. It needs read only access to a repository and write access to wiki. You can create variables for the pipeline. Set it as secret so that the value wouldn’t be visible.

Tasks I’ve used

A task is the building block for defining automation in a pipeline. A task is simply a packaged script or procedure that has been abstracted with a set of inputs.

Test

# Test
- task: DotNetCoreCLI@2
  displayName: Test Projects
  inputs:
    command: test
    projects: $(testProjects)
    publishTestResults: true
    arguments: '--no-build -c $(buildConfiguration) /p:Platform=x64'      

Build

# Install dotnet  
- task: UseDotNet@2        
  inputs:
    packageType: 'sdk'
    version: $(dotnetCliVersion)
    displayName: dotnet install

# Restore packages
- task: DotNetCoreCLI@2
  displayName: Restore Packages
  inputs:
    command: restore
    feedsToUse: select
    includeNuGetOrg: false
    projects: $(buildProjects)
    noCache: false

# Build          
- task: DotNetCoreCLI@2
  displayName: Build Projects
  inputs:
    projects: $(buildProjects)
    packDirectory: '$(build.artifactStagingDirectory)'
    arguments: '-c $(buildConfiguration) /p:Platform=x64 /p:Version=$(Build.BuildNumber) --no-restore'      

Publish and add it as artifact

# dotnet publish
- task: DotNetCoreCLI@2
  displayName: Publish Console Project
  inputs:
    command: publish
    publishWebProjects: false
    zipAfterPublish: false
    projects: $(cliProject)
    arguments: --no-build --no-restore -c Release /p:Platform=x64 /p:Version=$(Build.BuildNumber) -o "$(build.artifactStagingDirectory)\cli\"

# publics the artifact
- publish: $(build.artifactStagingDirectory)\cli
  displayName: Publish Artifact
  artifact: cli

Pack nuget package and publish to the feed

# pack nuget package
- task: NuGetCommand@2
  displayName: Pack
  condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/master'))
  inputs:
    command: 'pack'
    packagesToPack: '**/*.nuspec'
    versioningScheme: 'byBuildNumber'
    buildProperties: 'Platform=x64'

# Publish to the innovation release feed
- task: DotNetCoreCLI@2 
  displayName: Publish to Nuget Feed
  condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/master'))
  inputs:
    command: push
    packagesToPush: '$(build.artifactStagingDirectory)/*.nupkg'
    publishVstsFeed: ''