CI/CD Pipeline Patterns Master Class

Engineering-Grade Reference Manual for CI/CD Pipelines
A comprehensive guide to continuous integration and continuous deployment across major platforms

CI/CD Fundamentals
GitHub Actions
GitLab CI/CD
Jenkins
Azure DevOps
Deployment Strategies
Testing Automation
Secrets Management
Artifact Management
Pipeline Optimization
Security Best Practices
Monitoring & Observability
Real-World Examples
Troubleshooting
Best Practices

CI/CD Fundamentals

🔹 What is CI/CD?

Continuous Integration (CI):

Automatically build and test code changes
Merge code frequently to main branch
Catch bugs early in development
Maintain code quality standards

Continuous Delivery (CD):

Automatically prepare code for release
Deploy to staging/production environments
Ensure deployable state at all times
Reduce manual deployment steps

Continuous Deployment:

Automatically deploy to production
No manual intervention required
Every change goes through automated pipeline
Requires high confidence in testing

🔹 CI/CD Pipeline Stages

┌─────────────┐
│   Commit    │  Developer pushes code
└──────┬──────┘
       │
┌──────▼──────┐
│    Build    │  Compile, package application
└──────┬──────┘
       │
┌──────▼──────┐
│    Test     │  Unit, integration, e2e tests
└──────┬──────┘
       │
┌──────▼──────┐
│   Security  │  Vulnerability scanning
└──────┬──────┘
       │
┌──────▼──────┐
│   Deploy    │  Deploy to environment
└──────┬──────┘
       │
┌──────▼──────┐
│   Verify    │  Smoke tests, health checks
└─────────────┘

🔹 Key Concepts

Pipeline - Automated workflow from code to deployment
Job - Unit of work in a pipeline
Stage - Group of related jobs
Runner/Agent - Machine that executes pipeline jobs
Artifact - Files produced by pipeline (binaries, reports)
Trigger - Event that starts a pipeline (push, PR, schedule)
Environment - Deployment target (dev, staging, production)

GitHub Actions

🔹 Workflow Basics

✔️ Simple Workflow

# .github/workflows/ci.yml
name: CI Pipeline

on:
  push:
    branches: [ main, develop ]
  pull_request:
    branches: [ main ]

jobs:
  build:
    runs-on: ubuntu-latest
    
    steps:
      - name: Checkout code
        uses: actions/checkout@v4
      
      - name: Set up Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '18'
          cache: 'npm'
      
      - name: Install dependencies
        run: npm ci
      
      - name: Run tests
        run: npm test
      
      - name: Build application
        run: npm run build

✔️ Multi-Job Workflow

name: Full CI/CD Pipeline

on:
  push:
    branches: [ main ]

jobs:
  test:
    runs-on: ubuntu-latest
    
    steps:
      - uses: actions/checkout@v4
      
      - name: Set up Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '18'
      
      - name: Install dependencies
        run: npm ci
      
      - name: Run unit tests
        run: npm run test:unit
      
      - name: Run integration tests
        run: npm run test:integration
  
  build:
    needs: test
    runs-on: ubuntu-latest
    
    steps:
      - uses: actions/checkout@v4
      
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3
      
      - name: Login to Docker Hub
        uses: docker/login-action@v3
        with:
          username: ${{ secrets.DOCKER_USERNAME }}
          password: ${{ secrets.DOCKER_PASSWORD }}
      
      - name: Build and push
        uses: docker/build-push-action@v5
        with:
          context: .
          push: true
          tags: myapp:${{ github.sha }}
          cache-from: type=registry,ref=myapp:buildcache
          cache-to: type=registry,ref=myapp:buildcache,mode=max
  
  deploy:
    needs: build
    runs-on: ubuntu-latest
    environment: production
    
    steps:
      - name: Deploy to Kubernetes
        run: |
          kubectl set image deployment/myapp \
            myapp=myapp:${{ github.sha }} \
            --record

✔️ Matrix Builds

name: Matrix Build

on: [push]

jobs:
  test:
    runs-on: ${{ matrix.os }}
    
    strategy:
      matrix:
        os: [ubuntu-latest, windows-latest, macos-latest]
        node-version: [16, 18, 20]
        exclude:
          - os: macos-latest
            node-version: 16
    
    steps:
      - uses: actions/checkout@v4
      
      - name: Set up Node.js ${{ matrix.node-version }}
        uses: actions/setup-node@v4
        with:
          node-version: ${{ matrix.node-version }}
      
      - name: Install dependencies
        run: npm ci
      
      - name: Run tests
        run: npm test

✔️ Reusable Workflows

# .github/workflows/reusable-build.yml
name: Reusable Build

on:
  workflow_call:
    inputs:
      environment:
        required: true
        type: string
    secrets:
      docker-username:
        required: true
      docker-password:
        required: true

jobs:
  build:
    runs-on: ubuntu-latest
    
    steps:
      - uses: actions/checkout@v4
      
      - name: Build for ${{ inputs.environment }}
        run: npm run build:${{ inputs.environment }}

# .github/workflows/main.yml
name: Main Pipeline

on: [push]

jobs:
  build-staging:
    uses: ./.github/workflows/reusable-build.yml
    with:
      environment: staging
    secrets:
      docker-username: ${{ secrets.DOCKER_USERNAME }}
      docker-password: ${{ secrets.DOCKER_PASSWORD }}

✔️ Composite Actions

# .github/actions/setup-app/action.yml
name: 'Setup Application'
description: 'Setup Node.js and install dependencies'

inputs:
  node-version:
    description: 'Node.js version'
    required: false
    default: '18'

runs:
  using: 'composite'
  steps:
    - name: Set up Node.js
      uses: actions/setup-node@v4
      with:
        node-version: ${{ inputs.node-version }}
        cache: 'npm'
    
    - name: Install dependencies
      run: npm ci
      shell: bash
    
    - name: Cache build
      uses: actions/cache@v3
      with:
        path: |
          ~/.npm
          node_modules
        key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}

✔️ Using Composite Action

jobs:
  build:
    runs-on: ubuntu-latest
    
    steps:
      - uses: actions/checkout@v4
      
      - name: Setup application
        uses: ./.github/actions/setup-app
        with:
          node-version: '18'
      
      - name: Build
        run: npm run build

GitLab CI/CD

🔹 Pipeline Configuration

✔️ Basic Pipeline

# .gitlab-ci.yml
stages:
  - build
  - test
  - deploy

variables:
  DOCKER_IMAGE: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA

build:
  stage: build
  image: node:18
  script:
    - npm ci
    - npm run build
  artifacts:
    paths:
      - dist/
    expire_in: 1 hour
  cache:
    key: ${CI_COMMIT_REF_SLUG}
    paths:
      - node_modules/

test:unit:
  stage: test
  image: node:18
  script:
    - npm ci
    - npm run test:unit
  coverage: '/Coverage: \d+\.\d+/'
  artifacts:
    reports:
      junit: junit.xml
      coverage_report:
        coverage_format: cobertura
        path: coverage/cobertura-coverage.xml

test:integration:
  stage: test
  image: node:18
  services:
    - postgres:15
    - redis:7
  variables:
    POSTGRES_DB: testdb
    POSTGRES_USER: testuser
    POSTGRES_PASSWORD: testpass
  script:
    - npm ci
    - npm run test:integration

deploy:staging:
  stage: deploy
  image: alpine/k8s:latest
  script:
    - kubectl config use-context staging
    - kubectl set image deployment/myapp myapp=$DOCKER_IMAGE
    - kubectl rollout status deployment/myapp
  environment:
    name: staging
    url: https://staging.example.com
  only:
    - develop

deploy:production:
  stage: deploy
  image: alpine/k8s:latest
  script:
    - kubectl config use-context production
    - kubectl set image deployment/myapp myapp=$DOCKER_IMAGE
    - kubectl rollout status deployment/myapp
  environment:
    name: production
    url: https://example.com
  when: manual
  only:
    - main

✔️ Docker Build Pipeline

build:docker:
  stage: build
  image: docker:24
  services:
    - docker:24-dind
  variables:
    DOCKER_TLS_CERTDIR: "/certs"
  before_script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
  script:
    - docker build -t $DOCKER_IMAGE .
    - docker push $DOCKER_IMAGE
  tags:
    - docker

✔️ Multi-Project Pipeline

# Trigger downstream pipeline
trigger:microservice:
  stage: deploy
  trigger:
    project: team/microservice
    branch: main
    strategy: depend
  only:
    - main

✔️ Include External Configuration

include:
  - local: '/templates/.gitlab-ci-template.yml'
  - project: 'group/ci-templates'
    file: '/templates/docker-build.yml'
  - remote: 'https://example.com/ci-template.yml'

stages:
  - build
  - test
  - deploy

# Use included templates
build:
  extends: .docker-build-template

✔️ Dynamic Child Pipelines

generate:config:
  stage: build
  script:
    - python generate-pipeline.py > generated-config.yml
  artifacts:
    paths:
      - generated-config.yml

child:pipeline:
  stage: test
  trigger:
    include:
      - artifact: generated-config.yml
        job: generate:config
    strategy: depend

Jenkins

🔹 Declarative Pipeline

✔️ Basic Jenkinsfile

// Jenkinsfile
pipeline {
    agent any
    
    environment {
        DOCKER_IMAGE = "myapp:${env.BUILD_NUMBER}"
        REGISTRY = "docker.io"
    }
    
    stages {
        stage('Checkout') {
            steps {
                checkout scm
            }
        }
        
        stage('Build') {
            steps {
                sh 'npm ci'
                sh 'npm run build'
            }
        }
        
        stage('Test') {
            parallel {
                stage('Unit Tests') {
                    steps {
                        sh 'npm run test:unit'
                    }
                }
                stage('Integration Tests') {
                    steps {
                        sh 'npm run test:integration'
                    }
                }
            }
        }
        
        stage('Docker Build') {
            steps {
                script {
                    docker.build(DOCKER_IMAGE)
                }
            }
        }
        
        stage('Deploy to Staging') {
            when {
                branch 'develop'
            }
            steps {
                sh 'kubectl set image deployment/myapp myapp=${DOCKER_IMAGE}'
            }
        }
        
        stage('Deploy to Production') {
            when {
                branch 'main'
            }
            steps {
                input message: 'Deploy to production?', ok: 'Deploy'
                sh 'kubectl set image deployment/myapp myapp=${DOCKER_IMAGE}'
            }
        }
    }
    
    post {
        always {
            junit 'test-results/**/*.xml'
            cleanWs()
        }
        success {
            slackSend color: 'good', message: "Build ${env.BUILD_NUMBER} succeeded"
        }
        failure {
            slackSend color: 'danger', message: "Build ${env.BUILD_NUMBER} failed"
        }
    }
}

✔️ Advanced Pipeline with Agents

pipeline {
    agent none
    
    stages {
        stage('Build') {
            agent {
                docker {
                    image 'node:18'
                    args '-v $HOME/.npm:/root/.npm'
                }
            }
            steps {
                sh 'npm ci'
                sh 'npm run build'
                stash includes: 'dist/**', name: 'build-artifacts'
            }
        }
        
        stage('Test') {
            agent {
                kubernetes {
                    yaml '''
                        apiVersion: v1
                        kind: Pod
                        spec:
                          containers:
                          - name: node
                            image: node:18
                            command: ['cat']
                            tty: true
                          - name: postgres
                            image: postgres:15
                            env:
                            - name: POSTGRES_PASSWORD
                              value: testpass
                    '''
                }
            }
            steps {
                container('node') {
                    unstash 'build-artifacts'
                    sh 'npm test'
                }
            }
        }
        
        stage('Security Scan') {
            agent any
            steps {
                sh 'trivy image myapp:${BUILD_NUMBER}'
            }
        }
    }
}

✔️ Shared Library

// vars/buildDockerImage.groovy
def call(String imageName, String tag = 'latest') {
    sh """
        docker build -t ${imageName}:${tag} .
        docker push ${imageName}:${tag}
    """
}

// Jenkinsfile using shared library
@Library('my-shared-library') _

pipeline {
    agent any
    
    stages {
        stage('Build Docker Image') {
            steps {
                buildDockerImage('myapp', env.BUILD_NUMBER)
            }
        }
    }
}

✔️ Scripted Pipeline

node {
    try {
        stage('Checkout') {
            checkout scm
        }
        
        stage('Build') {
            sh 'npm ci'
            sh 'npm run build'
        }
        
        stage('Test') {
            parallel(
                'Unit Tests': {
                    sh 'npm run test:unit'
                },
                'Integration Tests': {
                    sh 'npm run test:integration'
                }
            )
        }
        
        stage('Deploy') {
            if (env.BRANCH_NAME == 'main') {
                input message: 'Deploy to production?'
                sh 'kubectl apply -f k8s/'
            }
        }
        
        currentBuild.result = 'SUCCESS'
    } catch (Exception e) {
        currentBuild.result = 'FAILURE'
        throw e
    } finally {
        // Cleanup
        cleanWs()
    }
}

Azure DevOps

🔹 Azure Pipelines

✔️ Basic Pipeline

# azure-pipelines.yml
trigger:
  branches:
    include:
      - main
      - develop

pool:
  vmImage: 'ubuntu-latest'

variables:
  buildConfiguration: 'Release'
  dockerImage: 'myapp:$(Build.BuildId)'

stages:
  - stage: Build
    jobs:
      - job: BuildJob
        steps:
          - task: NodeTool@0
            inputs:
              versionSpec: '18.x'
            displayName: 'Install Node.js'
          
          - script: |
              npm ci
              npm run build
            displayName: 'Build application'
          
          - task: PublishBuildArtifacts@1
            inputs:
              PathtoPublish: 'dist'
              ArtifactName: 'drop'
            displayName: 'Publish artifacts'

  - stage: Test
    dependsOn: Build
    jobs:
      - job: TestJob
        steps:
          - task: NodeTool@0
            inputs:
              versionSpec: '18.x'
          
          - script: npm ci
            displayName: 'Install dependencies'
          
          - script: npm test
            displayName: 'Run tests'
          
          - task: PublishTestResults@2
            inputs:
              testResultsFormat: 'JUnit'
              testResultsFiles: '**/test-results.xml'
            displayName: 'Publish test results'
          
          - task: PublishCodeCoverageResults@1
            inputs:
              codeCoverageTool: 'Cobertura'
              summaryFileLocation: '$(System.DefaultWorkingDirectory)/coverage/cobertura-coverage.xml'

  - stage: Deploy
    dependsOn: Test
    condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/main'))
    jobs:
      - deployment: DeployProduction
        environment: 'production'
        strategy:
          runOnce:
            deploy:
              steps:
                - task: Kubernetes@1
                  inputs:
                    connectionType: 'Kubernetes Service Connection'
                    kubernetesServiceEndpoint: 'k8s-production'
                    command: 'set'
                    arguments: 'image deployment/myapp myapp=$(dockerImage)'

✔️ Multi-Stage Docker Pipeline

stages:
  - stage: BuildDocker
    jobs:
      - job: BuildImage
        steps:
          - task: Docker@2
            displayName: 'Build Docker image'
            inputs:
              command: 'build'
              repository: 'myapp'
              dockerfile: 'Dockerfile'
              tags: |
                $(Build.BuildId)
                latest
          
          - task: Docker@2
            displayName: 'Push Docker image'
            inputs:
              command: 'push'
              repository: 'myapp'
              containerRegistry: 'dockerhub'
              tags: |
                $(Build.BuildId)
                latest
          
          - task: Docker@2
            displayName: 'Scan image for vulnerabilities'
            inputs:
              command: 'run'
              arguments: 'aquasec/trivy image myapp:$(Build.BuildId)'

✔️ Template Usage

# templates/build-template.yml
parameters:
  - name: nodeVersion
    type: string
    default: '18.x'

steps:
  - task: NodeTool@0
    inputs:
      versionSpec: ${{ parameters.nodeVersion }}
  
  - script: npm ci
    displayName: 'Install dependencies'
  
  - script: npm run build
    displayName: 'Build application'

# azure-pipelines.yml
stages:
  - stage: Build
    jobs:
      - job: BuildJob
        steps:
          - template: templates/build-template.yml
            parameters:
              nodeVersion: '18.x'

✔️ Release Pipeline

# Release pipeline with approvals
stages:
  - stage: DeployStaging
    jobs:
      - deployment: DeployToStaging
        environment: 'staging'
        strategy:
          runOnce:
            deploy:
              steps:
                - script: kubectl apply -f k8s/staging/
                  displayName: 'Deploy to staging'

  - stage: DeployProduction
    dependsOn: DeployStaging
    jobs:
      - deployment: DeployToProduction
        environment: 'production'  # Requires manual approval
        strategy:
          runOnce:
            preDeploy:
              steps:
                - script: echo "Running pre-deployment checks"
            deploy:
              steps:
                - script: kubectl apply -f k8s/production/
                  displayName: 'Deploy to production'
            postDeploy:
              steps:
                - script: echo "Running smoke tests"

Deployment Strategies

🔹 Blue-Green Deployment

✔️ Concept

┌─────────────┐
│   Router    │
└──────┬──────┘
       │
       ├─────────────┐
       │             │
┌──────▼──────┐ ┌───▼────────┐
│ Blue (v1.0) │ │Green (v2.0)│
│  Active     │ │   Idle     │
└─────────────┘ └────────────┘

After validation, switch router:

┌─────────────┐
│   Router    │
└──────┬──────┘
       │
       ├─────────────┐
       │             │
┌──────▼──────┐ ┌───▼────────┐
│ Blue (v1.0) │ │Green (v2.0)│
│   Idle      │ │  Active    │
└─────────────┘ └────────────┘

✔️ Kubernetes Implementation

# blue-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-blue
spec:
  replicas: 3
  selector:
    matchLabels:
      app: myapp
      version: blue
  template:
    metadata:
      labels:
        app: myapp
        version: blue
    spec:
      containers:
      - name: myapp
        image: myapp:v1.0
        ports:
        - containerPort: 8080

---
# green-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-green
spec:
  replicas: 3
  selector:
    matchLabels:
      app: myapp
      version: green
  template:
    metadata:
      labels:
        app: myapp
        version: green
    spec:
      containers:
      - name: myapp
        image: myapp:v2.0
        ports:
        - containerPort: 8080

---
# service.yaml
apiVersion: v1
kind: Service
metadata:
  name: myapp
spec:
  selector:
    app: myapp
    version: blue  # Switch to 'green' to activate new version
  ports:
  - port: 80
    targetPort: 8080

✔️ GitHub Actions Blue-Green

name: Blue-Green Deployment

on:
  push:
    branches: [ main ]

jobs:
  deploy:
    runs-on: ubuntu-latest
    
    steps:
      - uses: actions/checkout@v4
      
      - name: Determine current active version
        id: current
        run: |
          ACTIVE=$(kubectl get svc myapp -o jsonpath='{.spec.selector.version}')
          if [ "$ACTIVE" == "blue" ]; then
            echo "inactive=green" >> $GITHUB_OUTPUT
          else
            echo "inactive=blue" >> $GITHUB_OUTPUT
          fi
      
      - name: Deploy to inactive environment
        run: |
          kubectl set image deployment/myapp-${{ steps.current.outputs.inactive }} \
            myapp=myapp:${{ github.sha }}
          kubectl rollout status deployment/myapp-${{ steps.current.outputs.inactive }}
      
      - name: Run smoke tests
        run: |
          kubectl port-forward svc/myapp-${{ steps.current.outputs.inactive }} 8080:80 &
          sleep 5
          curl -f http://localhost:8080/health || exit 1
      
      - name: Switch traffic
        run: |
          kubectl patch svc myapp -p \
            '{"spec":{"selector":{"version":"${{ steps.current.outputs.inactive }}"}}}'
      
      - name: Verify deployment
        run: |
          sleep 10
          curl -f https://myapp.example.com/health

🔹 Canary Deployment

✔️ Concept

Initial: 100% v1.0

┌─────────────┐
│   Router    │
└──────┬──────┘
       │
       └─────────────┐
                     │
              ┌──────▼──────┐
              │  v1.0 (100%)│
              └─────────────┘

Canary: 90% v1.0, 10% v2.0

┌─────────────┐
│   Router    │
└──────┬──────┘
       │
       ├─────────────┬─────────────┐
       │             │             │
┌──────▼──────┐ ┌───▼────┐  ┌────▼────┐
│  v1.0 (90%) │ │v2.0(5%)│  │v2.0(5%) │
└─────────────┘ └────────┘  └─────────┘

Gradually increase v2.0 traffic

✔️ Kubernetes with Istio

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: myapp
spec:
  hosts:
  - myapp.example.com
  http:
  - match:
    - headers:
        canary:
          exact: "true"
    route:
    - destination:
        host: myapp
        subset: v2
  - route:
    - destination:
        host: myapp
        subset: v1
      weight: 90
    - destination:
        host: myapp
        subset: v2
      weight: 10

---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: myapp
spec:
  host: myapp
  subsets:
  - name: v1
    labels:
      version: v1
  - name: v2
    labels:
      version: v2

✔️ GitLab Canary Deployment

deploy:canary:
  stage: deploy
  script:
    - kubectl apply -f k8s/canary/
    - |
      # Start with 10% traffic
      kubectl patch virtualservice myapp --type merge -p '
      {
        "spec": {
          "http": [{
            "route": [
              {"destination": {"subset": "v1"}, "weight": 90},
              {"destination": {"subset": "v2"}, "weight": 10}
            ]
          }]
        }
      }'
  environment:
    name: production/canary
  when: manual

deploy:production:
  stage: deploy
  script:
    - |
      # Gradually increase to 100%
      for weight in 25 50 75 100; do
        kubectl patch virtualservice myapp --type merge -p "
        {
          \"spec\": {
            \"http\": [{
              \"route\": [
                {\"destination\": {\"subset\": \"v1\"}, \"weight\": $((100-weight))},
                {\"destination\": {\"subset\": \"v2\"}, \"weight\": ${weight}}
              ]
            }]
          }
        }"
        sleep 300  # Wait 5 minutes between increments
      done
  environment:
    name: production
  when: manual

🔹 Rolling Update

✔️ Kubernetes Rolling Update

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
spec:
  replicas: 10
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 2        # Max 2 extra pods during update
      maxUnavailable: 1  # Max 1 pod unavailable during update
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
      - name: myapp
        image: myapp:v2.0
        readinessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 5
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 15
          periodSeconds: 10

✔️ GitHub Actions Rolling Update

name: Rolling Update

on:
  push:
    branches: [ main ]

jobs:
  deploy:
    runs-on: ubuntu-latest
    
    steps:
      - name: Update deployment
        run: |
          kubectl set image deployment/myapp \
            myapp=myapp:${{ github.sha }} \
            --record
      
      - name: Monitor rollout
        run: |
          kubectl rollout status deployment/myapp --timeout=5m
      
      - name: Verify deployment
        run: |
          # Check all pods are ready
          kubectl wait --for=condition=ready pod \
            -l app=myapp \
            --timeout=5m
          
          # Run smoke tests
          kubectl run smoke-test --rm -i --restart=Never \
            --image=curlimages/curl -- \
            curl -f http://myapp/health
      
      - name: Rollback on failure
        if: failure()
        run: |
          kubectl rollout undo deployment/myapp
          kubectl rollout status deployment/myapp

Testing Automation

🔹 Unit Tests

✔️ GitHub Actions with Coverage

name: Unit Tests

on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    
    steps:
      - uses: actions/checkout@v4
      
      - name: Set up Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '18'
      
      - name: Install dependencies
        run: npm ci
      
      - name: Run unit tests
        run: npm run test:unit -- --coverage
      
      - name: Upload coverage to Codecov
        uses: codecov/codecov-action@v3
        with:
          files: ./coverage/lcov.info
          flags: unittests
          name: codecov-umbrella
      
      - name: Comment PR with coverage
        if: github.event_name == 'pull_request'
        uses: romeovs/lcov-reporter-action@v0.3.1
        with:
          lcov-file: ./coverage/lcov.info
          github-token: ${{ secrets.GITHUB_TOKEN }}

🔹 Integration Tests

✔️ GitLab with Services

test:integration:
  stage: test
  image: node:18
  services:
    - name: postgres:15
      alias: postgres
    - name: redis:7
      alias: redis
    - name: rabbitmq:3
      alias: rabbitmq
  
  variables:
    DATABASE_URL: "postgresql://testuser:testpass@postgres:5432/testdb"
    REDIS_URL: "redis://redis:6379"
    RABBITMQ_URL: "amqp://guest:guest@rabbitmq:5672"
    POSTGRES_DB: testdb
    POSTGRES_USER: testuser
    POSTGRES_PASSWORD: testpass
  
  before_script:
    - npm ci
    - npm run db:migrate
  
  script:
    - npm run test:integration
  
  artifacts:
    when: always
    reports:
      junit: test-results/integration/*.xml

🔹 End-to-End Tests

✔️ GitHub Actions with Playwright

name: E2E Tests

on:
  push:
    branches: [ main, develop ]

jobs:
  e2e:
    runs-on: ubuntu-latest
    
    steps:
      - uses: actions/checkout@v4
      
      - name: Set up Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '18'
      
      - name: Install dependencies
        run: npm ci
      
      - name: Install Playwright browsers
        run: npx playwright install --with-deps
      
      - name: Start application
        run: |
          npm run build
          npm run start &
          npx wait-on http://localhost:3000
      
      - name: Run E2E tests
        run: npm run test:e2e
      
      - name: Upload test results
        if: always()
        uses: actions/upload-artifact@v3
        with:
          name: playwright-report
          path: playwright-report/
          retention-days: 30

🔹 Performance Tests

✔️ Jenkins with k6

pipeline {
    agent any
    
    stages {
        stage('Deploy to Test Environment') {
            steps {
                sh 'kubectl apply -f k8s/test/'
                sh 'kubectl wait --for=condition=ready pod -l app=myapp --timeout=5m'
            }
        }
        
        stage('Performance Tests') {
            steps {
                script {
                    docker.image('grafana/k6:latest').inside {
                        sh '''
                            k6 run --out json=results.json \
                                --vus 100 \
                                --duration 5m \
                                performance-tests/load-test.js
                        '''
                    }
                }
            }
        }
        
        stage('Analyze Results') {
            steps {
                sh '''
                    # Check if p95 response time is under 500ms
                    P95=$(jq '.metrics.http_req_duration.values.p95' results.json)
                    if (( $(echo "$P95 > 500" | bc -l) )); then
                        echo "Performance degradation detected!"
                        exit 1
                    fi
                '''
            }
        }
    }
    
    post {
        always {
            archiveArtifacts artifacts: 'results.json'
            sh 'kubectl delete -f k8s/test/'
        }
    }
}

Secrets Management

🔹 GitHub Actions Secrets

name: Deploy with Secrets

on:
  push:
    branches: [ main ]

jobs:
  deploy:
    runs-on: ubuntu-latest
    
    steps:
      - uses: actions/checkout@v4
      
      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: us-east-1
      
      - name: Login to Docker Hub
        uses: docker/login-action@v3
        with:
          username: ${{ secrets.DOCKER_USERNAME }}
          password: ${{ secrets.DOCKER_PASSWORD }}
      
      - name: Deploy with environment-specific secrets
        env:
          DATABASE_URL: ${{ secrets.DATABASE_URL }}
          API_KEY: ${{ secrets.API_KEY }}
        run: |
          echo "DATABASE_URL=$DATABASE_URL" > .env
          echo "API_KEY=$API_KEY" >> .env
          kubectl create secret generic app-secrets --from-env-file=.env

🔹 HashiCorp Vault Integration

# GitHub Actions with Vault
name: Deploy with Vault

on:
  push:
    branches: [ main ]

jobs:
  deploy:
    runs-on: ubuntu-latest
    
    steps:
      - uses: actions/checkout@v4
      
      - name: Import Secrets from Vault
        uses: hashicorp/vault-action@v2
        with:
          url: https://vault.example.com
          token: ${{ secrets.VAULT_TOKEN }}
          secrets: |
            secret/data/production/database url | DATABASE_URL ;
            secret/data/production/api key | API_KEY
      
      - name: Deploy application
        env:
          DATABASE_URL: ${{ env.DATABASE_URL }}
          API_KEY: ${{ env.API_KEY }}
        run: |
          kubectl create secret generic app-secrets \
            --from-literal=database-url=$DATABASE_URL \
            --from-literal=api-key=$API_KEY

Artifact Management

🔹 GitHub Actions Artifacts

name: Build and Store Artifacts

on: [push]

jobs:
  build:
    runs-on: ubuntu-latest
    
    steps:
      - uses: actions/checkout@v4
      
      - name: Build application
        run: npm run build
      
      - name: Upload build artifacts
        uses: actions/upload-artifact@v3
        with:
          name: dist-${{ github.sha }}
          path: dist/
          retention-days: 30
      
      - name: Upload test results
        if: always()
        uses: actions/upload-artifact@v3
        with:
          name: test-results
          path: test-results/
  
  deploy:
    needs: build
    runs-on: ubuntu-latest
    
    steps:
      - name: Download artifacts
        uses: actions/download-artifact@v3
        with:
          name: dist-${{ github.sha }}
          path: dist/
      
      - name: Deploy artifacts
        run: |
          aws s3 sync dist/ s3://my-app-bucket/

🔹 Container Registry

name: Build and Push Container

on:
  push:
    tags:
      - 'v*'

jobs:
  build:
    runs-on: ubuntu-latest
    
    steps:
      - uses: actions/checkout@v4
      
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3
      
      - name: Login to GitHub Container Registry
        uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      
      - name: Extract metadata
        id: meta
        uses: docker/metadata-action@v5
        with:
          images: ghcr.io/${{ github.repository }}
          tags: |
            type=ref,event=branch
            type=semver,pattern={{version}}
            type=semver,pattern={{major}}.{{minor}}
            type=sha
      
      - name: Build and push
        uses: docker/build-push-action@v5
        with:
          context: .
          push: true
          tags: ${{ steps.meta.outputs.tags }}
          labels: ${{ steps.meta.outputs.labels }}
          cache-from: type=gha
          cache-to: type=gha,mode=max

Pipeline Optimization

🔹 Caching Strategies

✔️ GitHub Actions Cache

name: Optimized Build

on: [push]

jobs:
  build:
    runs-on: ubuntu-latest
    
    steps:
      - uses: actions/checkout@v4
      
      - name: Cache node modules
        uses: actions/cache@v3
        with:
          path: |
            ~/.npm
            node_modules
          key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}
          restore-keys: |
            ${{ runner.os }}-node-
      
      - name: Cache build output
        uses: actions/cache@v3
        with:
          path: dist
          key: ${{ runner.os }}-build-${{ github.sha }}
      
      - name: Install dependencies
        run: npm ci
      
      - name: Build
        run: npm run build

✔️ Docker Layer Caching

- name: Build with cache
  uses: docker/build-push-action@v5
  with:
    context: .
    push: true
    tags: myapp:latest
    cache-from: |
      type=registry,ref=myapp:buildcache
      type=gha
    cache-to: type=gha,mode=max

🔹 Parallel Execution

name: Parallel Jobs

on: [push]

jobs:
  test:
    strategy:
      matrix:
        test-suite: [unit, integration, e2e]
        node-version: [16, 18, 20]
      fail-fast: false
    
    runs-on: ubuntu-latest
    
    steps:
      - uses: actions/checkout@v4
      
      - name: Set up Node.js
        uses: actions/setup-node@v4
        with:
          node-version: ${{ matrix.node-version }}
      
      - name: Run ${{ matrix.test-suite }} tests
        run: npm run test:${{ matrix.test-suite }}

Security Best Practices

🔹 Dependency Scanning

name: Security Scan

on:
  push:
    branches: [ main ]
  pull_request:
  schedule:
    - cron: '0 0 * * 0'  # Weekly

jobs:
  security:
    runs-on: ubuntu-latest
    
    steps:
      - uses: actions/checkout@v4
      
      - name: Run Snyk security scan
        uses: snyk/actions/node@master
        env:
          SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }}
        with:
          args: --severity-threshold=high
      
      - name: Run npm audit
        run: npm audit --audit-level=moderate
      
      - name: Scan Docker image
        run: |
          docker build -t myapp:${{ github.sha }} .
          docker run --rm -v /var/run/docker.sock:/var/run/docker.sock \
            aquasec/trivy image myapp:${{ github.sha }}

🔹 SAST (Static Application Security Testing)

name: CodeQL Analysis

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

jobs:
  analyze:
    runs-on: ubuntu-latest
    
    permissions:
      security-events: write
    
    steps:
      - uses: actions/checkout@v4
      
      - name: Initialize CodeQL
        uses: github/codeql-action/init@v2
        with:
          languages: javascript
      
      - name: Autobuild
        uses: github/codeql-action/autobuild@v2
      
      - name: Perform CodeQL Analysis
        uses: github/codeql-action/analyze@v2

Monitoring & Observability

🔹 Pipeline Metrics

name: Pipeline with Metrics

on: [push]

jobs:
  build:
    runs-on: ubuntu-latest
    
    steps:
      - uses: actions/checkout@v4
      
      - name: Record start time
        id: start
        run: echo "time=$(date +%s)" >> $GITHUB_OUTPUT
      
      - name: Build application
        run: npm run build
      
      - name: Calculate build time
        run: |
          START=${{ steps.start.outputs.time }}
          END=$(date +%s)
          DURATION=$((END - START))
          echo "Build took ${DURATION} seconds"
          
          # Send to monitoring system
          curl -X POST https://metrics.example.com/api/metrics \
            -H "Content-Type: application/json" \
            -d "{
              \"metric\": \"build_duration\",
              \"value\": ${DURATION},
              \"tags\": {
                \"repo\": \"${{ github.repository }}\",
                \"branch\": \"${{ github.ref_name }}\"
              }
            }"

🔹 Deployment Notifications

- name: Notify Slack on deployment
  if: success()
  uses: slackapi/slack-github-action@v1
  with:
    payload: |
      {
        "text": "Deployment to production succeeded",
        "blocks": [
          {
            "type": "section",
            "text": {
              "type": "mrkdwn",
              "text": "*Deployment Successful* :white_check_mark:\n*Repository:* ${{ github.repository }}\n*Version:* ${{ github.sha }}\n*Deployed by:* ${{ github.actor }}"
            }
          }
        ]
      }
  env:
    SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK }}

Real-World Examples

🔹 Complete Node.js Application Pipeline

name: Full Stack CI/CD

on:
  push:
    branches: [ main, develop ]
  pull_request:
    branches: [ main ]

env:
  REGISTRY: ghcr.io
  IMAGE_NAME: ${{ github.repository }}

jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '18'
          cache: 'npm'
      - run: npm ci
      - run: npm run lint
  
  test:
    runs-on: ubuntu-latest
    needs: lint
    
    services:
      postgres:
        image: postgres:15
        env:
          POSTGRES_PASSWORD: testpass
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5
      
      redis:
        image: redis:7
        options: >-
          --health-cmd "redis-cli ping"
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5
    
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '18'
          cache: 'npm'
      
      - name: Install dependencies
        run: npm ci
      
      - name: Run unit tests
        run: npm run test:unit -- --coverage
      
      - name: Run integration tests
        env:
          DATABASE_URL: postgresql://postgres:testpass@localhost:5432/testdb
          REDIS_URL: redis://localhost:6379
        run: npm run test:integration
      
      - name: Upload coverage
        uses: codecov/codecov-action@v3
  
  build:
    runs-on: ubuntu-latest
    needs: test
    if: github.event_name == 'push'
    
    permissions:
      contents: read
      packages: write
    
    steps:
      - uses: actions/checkout@v4
      
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3
      
      - name: Log in to Container Registry
        uses: docker/login-action@v3
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      
      - name: Extract metadata
        id: meta
        uses: docker/metadata-action@v5
        with:
          images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
      
      - name: Build and push
        uses: docker/build-push-action@v5
        with:
          context: .
          push: true
          tags: ${{ steps.meta.outputs.tags }}
          labels: ${{ steps.meta.outputs.labels }}
          cache-from: type=gha
          cache-to: type=gha,mode=max
  
  deploy-staging:
    runs-on: ubuntu-latest
    needs: build
    if: github.ref == 'refs/heads/develop'
    environment:
      name: staging
      url: https://staging.example.com
    
    steps:
      - uses: actions/checkout@v4
      
      - name: Deploy to staging
        run: |
          kubectl config use-context staging
          kubectl set image deployment/myapp \
            myapp=${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.sha }}
          kubectl rollout status deployment/myapp
      
      - name: Run smoke tests
        run: |
          sleep 10
          curl -f https://staging.example.com/health
  
  deploy-production:
    runs-on: ubuntu-latest
    needs: build
    if: github.ref == 'refs/heads/main'
    environment:
      name: production
      url: https://example.com
    
    steps:
      - uses: actions/checkout@v4
      
      - name: Deploy to production
        run: |
          kubectl config use-context production
          kubectl set image deployment/myapp \
            myapp=${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.sha }}
          kubectl rollout status deployment/myapp
      
      - name: Run smoke tests
        run: |
          sleep 10
          curl -f https://example.com/health
      
      - name: Notify team
        if: always()
        uses: slackapi/slack-github-action@v1
        with:
          payload: |
            {
              "text": "Production deployment ${{ job.status }}",
              "blocks": [
                {
                  "type": "section",
                  "text": {
                    "type": "mrkdwn",
                    "text": "*Deployment Status:* ${{ job.status }}\n*Version:* ${{ github.sha }}\n*URL:* https://example.com"
                  }
                }
              ]
            }
        env:
          SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK }}

Troubleshooting

🔹 Common Issues

Pipeline Fails Intermittently

# Add retry logic
- name: Flaky test with retry
  uses: nick-invision/retry@v2
  with:
    timeout_minutes: 10
    max_attempts: 3
    command: npm run test:e2e

Slow Pipeline

# Use caching and parallel jobs
- name: Cache dependencies
  uses: actions/cache@v3
  with:
    path: node_modules
    key: ${{ runner.os }}-${{ hashFiles('**/package-lock.json') }}

# Run jobs in parallel
jobs:
  test-unit:
    runs-on: ubuntu-latest
  test-integration:
    runs-on: ubuntu-latest
  test-e2e:
    runs-on: ubuntu-latest

Out of Disk Space

# Clean up before build
- name: Free disk space
  run: |
    docker system prune -af
    sudo rm -rf /usr/share/dotnet
    sudo rm -rf /opt/ghc

Best Practices

✅ Pipeline Design

Keep pipelines fast - Optimize for speed
Fail fast - Run quick tests first
Use caching - Cache dependencies and build outputs
Parallel execution - Run independent jobs in parallel
Idempotent pipelines - Same input = same output
Version everything - Pin action/image versions
Secure secrets - Never log secrets
Monitor pipelines - Track metrics and failures

✅ Testing Strategy

Test pyramid - More unit tests, fewer e2e tests
Test in production-like environment
Automated smoke tests after deployment
Performance regression testing
Security scanning in pipeline

✅ Deployment Best Practices

Use deployment strategies - Blue-green, canary, rolling
Automated rollback - On health check failure
Gradual rollout - Don't deploy to all at once
Monitor after deployment - Watch metrics closely
Deployment windows - Deploy during low-traffic periods

Quick Reference Card

GitHub Actions

# Trigger on push and PR
on: [push, pull_request]

# Matrix build
strategy:
  matrix:
    os: [ubuntu, windows, macos]
    node: [16, 18, 20]

# Use secrets
${{ secrets.SECRET_NAME }}

# Cache dependencies
uses: actions/cache@v3

GitLab CI/CD

# Stages
stages: [build, test, deploy]

# Job with services
services:
  - postgres:15

# Artifacts
artifacts:
  paths: [dist/]
  expire_in: 1 hour

# Manual job
when: manual

Jenkins

// Declarative pipeline
pipeline {
    agent any
    stages {
        stage('Build') {
            steps {
                sh 'npm run build'
            }
        }
    }
}

// Parallel stages
parallel {
    stage('Test 1') { }
    stage('Test 2') { }
}

Azure DevOps

# Trigger
trigger:
  branches:
    include: [main]

# Multi-stage
stages:
  - stage: Build
    jobs:
      - job: BuildJob

# Template
- template: build-template.yml

🎓 CI/CD Pipeline Patterns Master Class Complete
This comprehensive guide covers everything from basic pipelines to advanced deployment strategies across all major CI/CD platforms. Use this as your reference for building production-ready automated workflows.

Track this guide

CI/CD Pipeline Patterns Master Class #

Table of Contents #

CI/CD Fundamentals #

🔹 What is CI/CD? #

🔹 CI/CD Pipeline Stages #

🔹 Key Concepts #

GitHub Actions #

🔹 Workflow Basics #

GitLab CI/CD #

🔹 Pipeline Configuration #

Jenkins #

🔹 Declarative Pipeline #

Azure DevOps #

🔹 Azure Pipelines #

Deployment Strategies #

🔹 Blue-Green Deployment #

🔹 Canary Deployment #

🔹 Rolling Update #

Testing Automation #

🔹 Unit Tests #

🔹 Integration Tests #

🔹 End-to-End Tests #

🔹 Performance Tests #

Secrets Management #

🔹 GitHub Actions Secrets #

🔹 HashiCorp Vault Integration #

Artifact Management #

🔹 GitHub Actions Artifacts #

🔹 Container Registry #

Pipeline Optimization #

🔹 Caching Strategies #

🔹 Parallel Execution #

Security Best Practices #

🔹 Dependency Scanning #

🔹 SAST (Static Application Security Testing) #

Monitoring & Observability #

🔹 Pipeline Metrics #

🔹 Deployment Notifications #

Real-World Examples #

🔹 Complete Node.js Application Pipeline #

Troubleshooting #

🔹 Common Issues #

Best Practices #

✅ Pipeline Design #

✅ Testing Strategy #

✅ Deployment Best Practices #

Quick Reference Card #

GitHub Actions #

GitLab CI/CD #

Jenkins #

Azure DevOps #

CI/CD Pipeline Patterns Master Class

Table of Contents

CI/CD Fundamentals

🔹 What is CI/CD?

🔹 CI/CD Pipeline Stages

🔹 Key Concepts

GitHub Actions

🔹 Workflow Basics

GitLab CI/CD

🔹 Pipeline Configuration

Jenkins

🔹 Declarative Pipeline

Azure DevOps

🔹 Azure Pipelines

Deployment Strategies

🔹 Blue-Green Deployment

🔹 Canary Deployment

🔹 Rolling Update

Testing Automation

🔹 Unit Tests

🔹 Integration Tests

🔹 End-to-End Tests

🔹 Performance Tests

Secrets Management

🔹 GitHub Actions Secrets

🔹 HashiCorp Vault Integration

Artifact Management

🔹 GitHub Actions Artifacts

🔹 Container Registry

Pipeline Optimization

🔹 Caching Strategies

🔹 Parallel Execution

Security Best Practices

🔹 Dependency Scanning

🔹 SAST (Static Application Security Testing)

Monitoring & Observability

🔹 Pipeline Metrics

🔹 Deployment Notifications

Real-World Examples

🔹 Complete Node.js Application Pipeline

Troubleshooting

🔹 Common Issues

Best Practices

✅ Pipeline Design

✅ Testing Strategy

✅ Deployment Best Practices

Quick Reference Card

GitHub Actions

GitLab CI/CD

Jenkins

Azure DevOps