Introduction
In this post I will be providing you with a Code Stream pipeline that is capable of exporting all pipelines in your Code Stream instance, committing them into a GitHub repository and pushing the commits back to GitHub.
Assumptions
I have made a few assumptions that you should note before we begin
- vRA Version – The version of vRA that this article relates to is 8.3 and may not work with different versions
- vRA 8.3 – You have a basic level of understanding of vRA, for example, how to navigate around and get to different places in the UI
- Code Stream Knowledge – You will need to have some knowledge of Code Stream, such as how to create a new pipeline, create Stages and Tasks, edit them and execute pipelines.
Prerequisites
You must meet the below prerequisites to follow along with this blog post.
- Code Stream Pipeline(s) – a number of pipelines created in Code Stream so you can test the backup pipeline.
- Docker Host – a docker host needs to be configured. The pipeline will be making use of a Docker Image created by Sam Perrin.
- GitHub Repository – you will need a GitHub repository. This can be private.
- GitHub API token – you will need a private key token to authenticate and access the remote GitHub repository
The Gotcha’s
- Variables – the pipeline uses variables to store sensitive information, such as credentials for accessing GitHub. You will need to manually recreate these since the Code Stream API doesn’t let us export them, even as placeholders without a value
- Pipeline YAML – the file will need to be edited to match your project, before it is imported (instructions below)
How It Works
The diagram below provides an overview of the process that the Pipeline goes through. For the JSON processing, I used the JQ library which made parsing the JSON responses from the REST API reasonably straightforward. Obviously there is some command line git in there and a small smattering of Bash wizardry with sed.
How Do I Use It?
- In Code Stream, select Pipelines from the menu
- Click the Import button
- Select the option to Create a new blueprint
- Copy and paste in the code below.
- Replace the <<PROJECT_NAME>> entry with the name of the Project in Code Stream you will be associating this pipeline with
- Click Import
- Open the pipeline and set values against all of the Inputs
- Create the Variables as shown in the below table
- Run it!
Pipeline Variables
Variable Name | Type | Description |
---|---|---|
Github Access Token | SECRET | Your GitHub access token |
vRA_Obtain_Token_Body | SECRET | Username and password for accessing vRA in JSON format Example: {"username":"administrator","password":"L3tM3!n!"} |
--- project: <<PROJECT_NAME>> kind: PIPELINE name: AutomationPro Backup Pipelines icon: organization,left, is-info enabled: true description: provided by Automationpro.co.uk concurrency: 10 options: [ DOCKER_TRIGGER] input: GITHUB_EMAIL: '' GITHUB_TARGET_REPOSITORY: '' GITHUB_USERNAME: '' REPO_FOLDER: repository VRA_SERVER: '' _inputMeta: GITHUB_EMAIL: mandatory: true description: Email address associated with your Git account (i.e yourname@gmail.com) GITHUB_USERNAME: mandatory: true description: GitHub username (i.e. anonuser) VRA_SERVER: mandatory: true description: The vRA Server or a node in vRA Cluster FQDN (i.e. vra1.server.local) REPO_FOLDER: mandatory: true description: The local repository to work in GITHUB_TARGET_REPOSITORY: mandatory: true description: GitHub repository remote address (i.e. github.com/anonuser/PipeLineBackup.git) workspace: endpoint: aprodkr1 - docker image: samperrin/alpine-codestream-ci:latest registry: '' path: '' autoCloneForTrigger: true limits: memory: 512 cpu: 1.0 stageOrder: - Initialization stages: Initialization: taskOrder: - Obtain vRA Token - Obtain Pipelines - Export Pipeline tasks: Export Pipeline: type: CI input: steps: - '#!/bin/bash' - git config --global user.email "${input.GITHUB_EMAIL}" - git config --global user.name "${input.GITHUB_USERNAME}" - git clone https://${var.Github Access Token}@${input.GITHUB_TARGET_REPOSITORY} ${input.REPO_FOLDER} - cd ${input.REPO_FOLDER} - '' - cat <<'EOF' >> /tmp/response.json - ${Initialization.Obtain Pipelines.output.responseBody} - EOF - '' - jq -r '.documents | .[] | "/pipeline/api/export?pipeline=" + .name + "&project=" + .project' /tmp/response.json | - '' - while IFS= read val - do - ' uriPart=`echo "$val"|sed ''s/ /%20/g''`' - ' filename=$(echo $uriPart| cut -d''='' -f 2 | cut -d''&'' -f 1)' - ' ' - ' curl -o $filename.yaml -k -H "Authorization: Bearer ${Initialization.Obtain vRA Token.output.responseJson.access_token}" https://${input.VRA_SERVER}$uriPart' - ' ' - ' git add $filename.yaml' - '' - ' gitStatus=$(git status --porcelain=v1 2>/dev/null | wc -l)' - ' if [ $gitStatus == 1 ]; then' - ' git commit -m "$filename commit $(date +%s))";' - ' git push https://${var.Github Access Token}@${input.GITHUB_TARGET_REPOSITORY}' - ' fi' - done export: [ ] artifacts: [ ] process: [ ] Obtain vRA Token: type: REST input: action: post url: https://${input.VRA_SERVER}/csp/gateway/am/api/login?access_token headers: Accept: application/json Content-Type: application/json payload: ${var.vRA_Obtain_Token_Body} Obtain Pipelines: type: REST input: action: get url: https://${input.VRA_SERVER}/pipeline/api/pipelines headers: Accept: application/json Content-Type: application/json Authorization: Bearer ${Initialization.Obtain vRA Token.output.responseJson.access_token} payload: ''