Introduction

In this post I will be providing you with a Code Stream pipeline that is capable of exporting all pipelines in your Code Stream instance, committing them into a GitHub repository and pushing the commits back to GitHub.

Assumptions

I have made a few assumptions that you should note before we begin

• vRA Version – The version of vRA that this article relates to is 8.3 and may not work with different versions
• vRA 8.3 – You have a basic level of understanding of vRA, for example, how to navigate around and get to different places in the UI
• Code Stream Knowledge – You will need to have some knowledge of Code Stream, such as how to create a new pipeline, create Stages and Tasks, edit them and execute pipelines.

Prerequisites

You must meet the below prerequisites to follow along with this blog post.

• Code Stream Pipeline(s) – a number of pipelines created in Code Stream so you can test the backup pipeline.
• Docker Host – a docker host needs to be configured. The pipeline will be making use of a Docker Image created by Sam Perrin.
• GitHub Repository – you will need a GitHub repository. This can be private.
• GitHub API token – you will need a private key token to authenticate and access the remote GitHub repository

The Gotcha’s

• Variables – the pipeline uses variables to store sensitive information, such as credentials for accessing GitHub. You will need to manually recreate these since the Code Stream API doesn’t let us export them, even as placeholders without a value
• Pipeline YAML – the file will need to be edited to match your project, before it is imported (instructions below)

How It Works

The diagram below provides an overview of the process that the Pipeline goes through. For the JSON processing, I used the JQ library which made parsing the JSON responses from the REST API reasonably straightforward. Obviously there is some command line git in there and a small smattering of Bash wizardry with sed.

How Do I Use It?

1. In Code Stream, select Pipelines from the menu
2. Click the Import button
3. Select the option to Create a new blueprint
4. Copy and paste in the code below.
5. Replace the <<PROJECT_NAME>> entry with the name of the Project in Code Stream you will be associating this pipeline with
6. Click Import
7. Open the pipeline and set values against all of the Inputs
8. Create the Variables as shown in the below table
9. Run it!

Pipeline Variables

---
project: <<PROJECT_NAME>>
kind: PIPELINE
name: AutomationPro Backup Pipelines
icon: organization,left, is-info
enabled: true
description: provided by Automationpro.co.uk
concurrency: 10
options: [
  DOCKER_TRIGGER]
input:
  GITHUB_EMAIL: ''
  GITHUB_TARGET_REPOSITORY: ''
  GITHUB_USERNAME: ''
  REPO_FOLDER: repository
  VRA_SERVER: ''
_inputMeta:
  GITHUB_EMAIL:
    mandatory: true
    description: Email address associated with your Git account (i.e yourname@gmail.com)
  GITHUB_USERNAME:
    mandatory: true
    description: GitHub username (i.e. anonuser)
  VRA_SERVER:
    mandatory: true
    description: The vRA Server or a node in vRA Cluster FQDN (i.e. vra1.server.local)
  REPO_FOLDER:
    mandatory: true
    description: The local repository to work in
  GITHUB_TARGET_REPOSITORY:
    mandatory: true
    description: GitHub repository remote address (i.e. github.com/anonuser/PipeLineBackup.git)
workspace:
  endpoint: aprodkr1 - docker
  image: samperrin/alpine-codestream-ci:latest
  registry: ''
  path: ''
  autoCloneForTrigger: true
  limits:
    memory: 512
    cpu: 1.0
stageOrder:
- Initialization
stages:
  Initialization:
    taskOrder:
    - Obtain vRA Token
    - Obtain Pipelines
    - Export Pipeline
    tasks:
      Export Pipeline:
        type: CI
        input:
          steps:
          - '#!/bin/bash'
          - git config --global user.email "${input.GITHUB_EMAIL}"
          - git config --global user.name "${input.GITHUB_USERNAME}"
          - git clone https://${var.Github Access Token}@${input.GITHUB_TARGET_REPOSITORY} ${input.REPO_FOLDER}
          - cd ${input.REPO_FOLDER}
          - ''
          - cat <<'EOF' >> /tmp/response.json
          - ${Initialization.Obtain Pipelines.output.responseBody}
          - EOF
          - ''
          - jq -r '.documents | .[] | "/pipeline/api/export?pipeline=" + .name + "&project=" + .project'
            /tmp/response.json |
          - ''
          - while IFS= read val
          - do
          - '  uriPart=`echo "$val"|sed ''s/ /%20/g''`'
          - '  filename=$(echo $uriPart| cut -d''='' -f 2 | cut -d''&'' -f 1)'
          - '  '
          - '  curl -o $filename.yaml -k -H "Authorization: Bearer ${Initialization.Obtain vRA Token.output.responseJson.access_token}"
            https://${input.VRA_SERVER}$uriPart'
          - '  '
          - '  git add $filename.yaml'
          - ''
          - '  gitStatus=$(git status --porcelain=v1 2>/dev/null | wc -l)'
          - '  if [ $gitStatus == 1 ]; then'
          - '    git commit -m "$filename commit $(date +%s))";'
          - '    git push https://${var.Github Access Token}@${input.GITHUB_TARGET_REPOSITORY}'
          - '  fi'
          - done
          export: [
            ]
          artifacts: [
            ]
          process: [
            ]
      Obtain vRA Token:
        type: REST
        input:
          action: post
          url: https://${input.VRA_SERVER}/csp/gateway/am/api/login?access_token
          headers:
            Accept: application/json
            Content-Type: application/json
          payload: ${var.vRA_Obtain_Token_Body}
      Obtain Pipelines:
        type: REST
        input:
          action: get
          url: https://${input.VRA_SERVER}/pipeline/api/pipelines
          headers:
            Accept: application/json
            Content-Type: application/json
            Authorization: Bearer ${Initialization.Obtain vRA Token.output.responseJson.access_token}
          payload: ''