Session 3: CI/CD — The Backbone

What is CI/CD?

CI/CD is the automated pipeline that takes your code from commit to production. It’s the backbone of DevOps — without it, everything else falls apart.

CI — Continuous Integration

Every time someone pushes code, it gets automatically built and tested against the main codebase.

Developer pushes code → Build → Run tests → ✅ Pass or ❌ Fail

The rules of CI:

• Merge to main frequently (at least daily)
• Automated tests run on every push
• If tests fail, fix it immediately — a broken main blocks everyone
• The main branch must always be in a working state

Without CI: A developer works for a week, pushes a massive change. It breaks everything. Nobody knows which of the 50 changes caused it.

With CI: A developer pushes 5 small changes a day. Each is tested. If something breaks, you know exactly which change caused it.

CD — Continuous Delivery vs Continuous Deployment

These two terms sound similar but mean different things:

Code → Build → Test → [Delivery: ready to deploy] → [Deployment: auto-deployed]
                              ↑ human gate              ↑ no gate

Most companies start with Continuous Delivery and evolve to Continuous Deployment as confidence grows.

Key insight: Deployment should be boring. If deploying is scary, you’re not deploying often enough. The more you deploy, the smaller each change is, the less risky it becomes.

Why CI/CD Matters for Reliability

Smaller changes = smaller blast radius. If a deploy with 5 lines of code breaks something, it’s easy to find and fix. If a deploy with 5,000 lines breaks something, good luck.

Jenkins — From Recap to Practical Automation

Jenkins is the veteran CI/CD tool. Many teams still use it — including ours, where it runs the chugli-qa test suite against chugli.ai and preview.chugli.ai. The rest of this section is a practical tour: what Jenkins actually consists of, the admin menus you’ll touch, how a real chugli-qa pipeline looks, and how to stop copy-pasting Jenkinsfiles by using a shared library.

Declarative Pipeline (Jenkinsfile)

pipeline {
    agent any

    stages {
        stage('Build') {
            steps {
                sh 'npm install'
            }
        }
        stage('Test') {
            steps {
                sh 'npm test'
            }
        }
        stage('Deploy') {
            when {
                branch 'main'
            }
            steps {
                sh './deploy.sh'
            }
        }
    }

    post {
        failure {
            mail to: '[email protected]',
                 subject: "Build Failed: ${env.JOB_NAME}",
                 body: "Check: ${env.BUILD_URL}"
        }
    }
}

Key concepts:

• Pipeline — the entire build/test/deploy process
• Stage — a logical group of steps (Build, Test, Deploy)
• Steps — individual tasks within a stage
• Agent — where the pipeline runs
• Post — actions after pipeline completes (notify on failure)

Jenkins vs Modern CI/CD

Jenkins Anatomy in 60 Seconds

You’ll spend an hour clicking around the Manage Jenkins menu and feel lost unless you know what’s actually under the hood.

                    ┌──────────────────────┐
                    │   Jenkins Controller │
                    │  (the web UI + brain)│
                    │                      │
                    │   $JENKINS_HOME/     │
                    │   ├── jobs/          │
                    │   ├── plugins/       │
                    │   ├── credentials.xml│
                    │   └── users/         │
                    └──────────┬───────────┘
                               │ dispatches builds
                ┌──────────────┼──────────────┐
                ▼              ▼              ▼
         ┌──────────┐   ┌──────────┐   ┌──────────┐
         │ Agent 1  │   │ Agent 2  │   │ Agent 3  │
         │ linux/x64│   │ docker   │   │ playwright│
         └──────────┘   └──────────┘   └──────────┘

Key intuition: the controller knows nothing about Python or Playwright. It hands a script to an agent and says “run this, tell me what happened.” The agent’s environment is what matters.

Admin Essentials You’ll Actually Touch

You don’t need to be a Jenkins admin, but you’ll click these menus regularly. Here’s what each one does.

Manage Jenkins → Plugins

Plugins make Jenkins useful. For chugli-qa, you’d want:

Rule: install only plugins you actively use. Each one is attack surface and an upgrade burden.

Manage Jenkins → Credentials

Where secrets live so they never appear in code or logs. For chugli-qa:

Never put any of these in Jenkinsfile. They go here once, and pipelines reference them by ID.

Manage Jenkins → Users / Security

• Matrix Authorization Strategy — table of users vs permissions.
• Default (“anyone authenticated can do anything”) is fine for an internal Jenkins behind a VPN, not for one exposed to the internet.
• Sane defaults: dev team = full access, QA team = build/cancel jobs but not configure, viewers = read-only.

Manage Jenkins → Nodes (Agents)

• Built-in node — runs on the controller itself. Don’t run builds here. A bad build can break Jenkins.
• Add an agent when you need: a different OS (Windows, macOS for iOS), a specific environment (Playwright browsers, GPU, Python version), or the controller is overloaded.
• Two ways to provide agents — a long-lived VM/container that stays connected, or a Docker container spun up per build (next section).

Concept: Docker Agents — Skip the Server, Use a Container

You could maintain a Linux VM as your linux-pytest agent — install Python, pip, Chromium, keep it patched, hope it doesn’t drift. That’s the old way.

The modern way: let Jenkins spin up a fresh Docker container for each build, then throw it away. The image is your environment definition. No drift, no patching, no “works on agent A, fails on agent B.”

How it looks in a Jenkinsfile

Two patterns. Pick whichever fits.

Pattern A — per-pipeline image (the whole pipeline runs inside one container):

pipeline {
    agent {
        docker {
            image 'python:3.11-slim'
            args '-u root'   // run as root inside the container
        }
    }
    stages {
        stage('Test') {
            steps {
                sh 'pip install -r requirements.txt && pytest'
            }
        }
    }
}

When this build runs, Jenkins:

1. Finds a host with Docker installed (still an “agent” in the Jenkins sense, but it just runs docker)
2. Pulls python:3.11-slim if not cached
3. Runs the whole pipeline inside that container
4. Removes the container when done

Pattern B — per-stage image (different stages use different containers):

pipeline {
    agent none

    stages {
        stage('Unit tests') {
            agent { docker { image 'python:3.11-slim' } }
            steps { sh 'pip install -r requirements.txt && pytest -m api' }
        }

        stage('E2E tests') {
            agent { docker { image 'mcr.microsoft.com/playwright/python:v1.45.0-jammy' } }
            steps { sh 'pip install -r requirements.txt && pytest -m e2e' }
        }
    }
}

The unit-test stage gets a tiny Python image. The E2E stage gets the full Playwright image with browsers pre-installed. No need to maintain either environment by hand.

Why this is better for chugli-qa

What you still need

You’re not running zero machines — you need at least one host with Docker installed that Jenkins can talk to. That host is the agent; it just delegates the actual work to short-lived containers. For a small team, the controller and the Docker host can even be the same machine (with care).

When to stick with a real VM agent

• You need GPU access for ML inference tests
• You need to test installers that touch the OS itself
• You need Windows or macOS (Docker on those is awkward in CI)

For everything else — including chugli-qa’s pytest + Playwright setup — Docker agents are the default in 2026.

A Real chugli-qa Pipeline

Here’s a working chugli-qa Jenkinsfile. Drop it at the root of the repo, point Jenkins at it, you have CI.

// chugli-qa/Jenkinsfile
pipeline {
    agent { label 'linux-pytest' }

    parameters {
        choice(
            name: 'SUITE',
            choices: ['api', 'data', 'e2e', 'all'],
            description: 'Which test suite to run'
        )
        string(
            name: 'BASE_URL',
            defaultValue: 'https://preview.chugli.ai',
            description: 'App URL to test against'
        )
    }

    triggers {
        cron('H 2 * * *')  // nightly at ~2am
    }

    options {
        timeout(time: 30, unit: 'MINUTES')
        ansiColor('xterm')
        timestamps()
    }

    environment {
        CHUGLI_BASE_URL = "${params.BASE_URL}"
    }

    stages {
        stage('Checkout') {
            steps {
                checkout scm
            }
        }

        stage('Setup venv') {
            steps {
                sh '''
                    python3 -m venv venv
                    . venv/bin/activate
                    pip install --quiet -r requirements.txt
                '''
            }
        }

        stage('Run tests') {
            steps {
                withCredentials([
                    string(credentialsId: 'chugli-qa-mongo-uri',
                           variable: 'MONGODB_URI'),
                    usernamePassword(credentialsId: 'chugli-qa-test-user',
                                     usernameVariable: 'TEST_USER_EMAIL',
                                     passwordVariable: 'TEST_USER_PASSWORD')
                ]) {
                    sh '''
                        . venv/bin/activate
                        if [ "$SUITE" = "all" ]; then
                            pytest --html=reports/report.html --self-contained-html
                        else
                            pytest -m $SUITE --html=reports/report.html --self-contained-html
                        fi
                    '''
                }
            }
        }
    }

    post {
        always {
            archiveArtifacts artifacts: 'reports/**', allowEmptyArchive: true
            publishHTML(target: [
                reportDir: 'reports',
                reportFiles: 'report.html',
                reportName: 'pytest report',
                keepAll: true
            ])
        }
        failure {
            withCredentials([string(credentialsId: 'discord-qa-webhook',
                                    variable: 'WEBHOOK')]) {
                sh '''
                    curl -X POST -H "Content-Type: application/json" \
                      -d "{\\"content\\": \\"chugli-qa #${BUILD_NUMBER} failed: ${BUILD_URL}\\"}" \
                      $WEBHOOK
                '''
            }
        }
    }
}

Reading it line-by-line

Multibranch Pipeline

A single Jenkinsfile job is fine for one branch. But chugli-qa has multiple active branches: main, feature/safety-tests, chugli-173-bugs. You want each to run its own tests.

You could create one job per branch by hand. Don’t. Use a Multibranch Pipeline.

What it does

1. Point it at the chugli-qa Git repo
2. It scans the repo and finds every branch and every PR
3. For each one with a Jenkinsfile at the root, it auto-creates a sub-job
4. When you push to a branch, that branch’s sub-job runs
5. When you delete a branch, the sub-job disappears

Setup (once)

New Item → Multibranch Pipeline → "chugli-qa"
  Branch Sources:
    Git
      Repository: [email protected]:dhustlerz/chugli-qa.git
      Credentials: github-pat-chugli-qa
      Behaviors: Discover branches, Discover PRs from origin
  Build Configuration:
    Mode: by Jenkinsfile
    Script Path: Jenkinsfile
  Scan Multibranch Pipeline Triggers:
    Periodically if not otherwise run: 5 minutes

What you get

chugli-qa/
├── main                       PASS  #128 — passed (2m 14s)
├── feature/safety-tests       RUN   #4   — running
├── chugli-173-bugs            FAIL  #11  — failed
└── PR-42 (from forked PR)     PASS  #2   — passed

Click any branch to see its build history. Push a commit, the build runs. Open a PR, a build runs against the PR head. Merge or close the PR, the sub-job vanishes.

This is how 95% of teams use Jenkins for app repos. Set it up once, forget it.

Shared Libraries — The “Why”

Imagine in 6 months we have three QA repos:

• chugli-qa — pytest + Playwright for chugli.ai
• ad-automation — pytest for the ad pipeline
• shaadinabanayo-qa — pytest for shaadinabanayo

Each one needs the same scaffolding: set up a Python venv, install requirements, run pytest with HTML reporting, archive the report, notify Discord on failure.

If we copy-paste the Jenkinsfile, we now maintain three copies. Fix a bug in the notify code? Three PRs. Switch from Discord to Slack? Three PRs.

Shared libraries let you write that scaffolding once and consume it from every Jenkinsfile.

Before — duplicated Jenkinsfile (40+ lines per repo)

pipeline {
    agent { label 'linux-pytest' }
    stages {
        stage('Setup') {
            steps {
                sh 'python3 -m venv venv && . venv/bin/activate && pip install -r requirements.txt'
            }
        }
        stage('Test') {
            steps {
                sh '. venv/bin/activate && pytest --html=reports/report.html --self-contained-html'
            }
        }
    }
    post {
        always { archiveArtifacts 'reports/**' }
        failure {
            withCredentials([string(credentialsId: 'discord-qa-webhook', variable: 'WEBHOOK')]) {
                sh 'curl -X POST -d ... $WEBHOOK'
            }
        }
    }
}

After — using a shared library (5 lines per repo)

@Library('dhustlerz-pipeline-library@main') _

runPytest(
    suite: 'api',
    notifyChannel: 'qa-alerts'
)

That’s the whole Jenkinsfile. All the setup, secret-handling, archiving, notifying — moved into the library. Three repos, one source of truth.

Writing Your Own Library Step

A shared library is a Git repo with a specific folder structure. Jenkins clones it and exposes its functions to every Jenkinsfile.

File layout

dhustlerz-pipeline-library/
├── vars/
│   └── runPytest.groovy        ← exposed as the global function `runPytest()`
├── src/
│   └── com/
│       └── dhustlerz/
│           └── Notify.groovy   ← reusable class
└── README.md

vars/runPytest.groovy — the custom step

// vars/runPytest.groovy
import com.dhustlerz.Notify

def call(Map config = [:]) {
    def suite = config.suite ?: 'all'
    def channel = config.notifyChannel ?: 'qa-alerts'

    pipeline {
        agent { label 'linux-pytest' }

        options {
            timeout(time: 30, unit: 'MINUTES')
            timestamps()
        }

        stages {
            stage('Setup venv') {
                steps {
                    sh '''
                        python3 -m venv venv
                        . venv/bin/activate
                        pip install --quiet -r requirements.txt
                    '''
                }
            }

            stage('Run tests') {
                steps {
                    withCredentials([
                        string(credentialsId: 'chugli-qa-mongo-uri',
                               variable: 'MONGODB_URI')
                    ]) {
                        sh """
                            . venv/bin/activate
                            pytest -m ${suite} --html=reports/report.html --self-contained-html
                        """
                    }
                }
            }
        }

        post {
            always {
                archiveArtifacts artifacts: 'reports/**', allowEmptyArchive: true
            }
            failure {
                script {
                    new Notify(this).discord(channel,
                        "${env.JOB_NAME} #${env.BUILD_NUMBER} failed: ${env.BUILD_URL}")
                }
            }
        }
    }
}

src/com/dhustlerz/Notify.groovy — the helper class

// src/com/dhustlerz/Notify.groovy
package com.dhustlerz

class Notify implements Serializable {
    def steps

    Notify(steps) { this.steps = steps }

    void discord(String channel, String message) {
        def credId = "discord-${channel}-webhook"
        steps.withCredentials([
            steps.string(credentialsId: credId, variable: 'WEBHOOK')
        ]) {
            steps.sh """
                curl -X POST -H "Content-Type: application/json" \
                  -d '{"content": "${message}"}' \
                  \$WEBHOOK
            """
        }
    }
}

Registering the library in Jenkins (once)

Manage Jenkins → System → Global Pipeline Libraries
  Name: dhustlerz-pipeline-library
  Default version: main
  Retrieval method: Modern SCM → Git
  Project repository: [email protected]:dhustlerz/jenkins-shared-library.git
  Credentials: github-pat

After this, any Jenkinsfile in any repo can do:

@Library('dhustlerz-pipeline-library@main') _
runPytest(suite: 'api', notifyChannel: 'qa-alerts')

Practical Patterns You’ll Use Over and Over

1. Parameterized jobs — let humans pick options before running

parameters {
    choice(name: 'SUITE', choices: ['api', 'data', 'e2e', 'all'], description: 'Test suite')
    string(name: 'BASE_URL', defaultValue: 'https://preview.chugli.ai', description: 'Target URL')
    booleanParam(name: 'DRY_RUN', defaultValue: false, description: 'Skip DB writes')
}

The “Build” button becomes “Build with Parameters” — a form appears. Use it for “run this on demand against a specific environment.”

2. Cron triggers — schedule recurring runs

triggers {
    cron('H 2 * * *')        // ~2am every day
    cron('H/15 * * * *')     // every 15 minutes
    cron('H 6 * * 1-5')      // ~6am weekdays only
}

The H (“hash”) spreads load — Jenkins picks a stable random minute per job, so you don’t get a thundering herd at exactly 02:00.

3. Credentials — never put secrets in the file

withCredentials([
    string(credentialsId: 'chugli-qa-mongo-uri', variable: 'MONGODB_URI'),
    usernamePassword(credentialsId: 'chugli-qa-test-user',
                     usernameVariable: 'EMAIL',
                     passwordVariable: 'PASSWORD')
]) {
    sh 'pytest'  // env vars are set, masked in logs
}

If a secret accidentally appears in a build log, Jenkins replaces it with **** — but only if you used withCredentials. Hardcoded values are not masked.

4. Artifact archiving — keep the report

archiveArtifacts artifacts: 'reports/**', allowEmptyArchive: true, fingerprint: true
publishHTML target: [reportDir: 'reports', reportFiles: 'report.html', reportName: 'pytest']

archiveArtifacts saves files into the build (downloadable). publishHTML makes the report viewable in the build sidebar — much nicer for non-engineers.

Try This on Your Own

Three exercises, ~30 minutes total, on the existing Jenkins instance.

Exercise 1 — Single pipeline job (10 min)

1. Log into Jenkins, New Item → Pipeline → “chugli-qa-manual”
2. In the Pipeline section, choose Pipeline script from SCM → Git
3. Repo: chugli-qa, Branch: main, Script Path: Jenkinsfile
4. (You may need to create the Jenkinsfile in chugli-qa first — copy from the example above)
5. Save → Build Now → watch it run

Goal: see a build go from queued → running → success/failure, click into the logs.

Exercise 2 — Convert to multibranch (10 min)

1. Delete the job from Exercise 1
2. New Item → Multibranch Pipeline → “chugli-qa”
3. Configure with the multibranch settings shown above
4. Save → Jenkins scans the repo
5. Push a commit to a feature branch → see it auto-build

Goal: confirm that pushing to a branch automatically triggers a build for that branch.

Exercise 3 — Extract one step into a shared library (10 min)

1. Create a Git repo jenkins-shared-library with vars/runPytest.groovy containing the example above
2. Register it under Manage Jenkins → System → Global Pipeline Libraries
3. Replace your chugli-qa Jenkinsfile with the 5-line version using @Library
4. Push, watch the same build pass

Goal: experience the “wait, my whole Jenkinsfile is 5 lines and the logic lives somewhere reusable” moment.

GitHub Actions — The Modern Approach

GitHub Actions is CI/CD built directly into GitHub. No separate server to manage.

Core Concepts

A Real Workflow File

# .github/workflows/ci.yml
name: CI Pipeline

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: 'npm'
      - run: npm ci
      - run: npm run lint

  test:
    runs-on: ubuntu-latest
    needs: lint
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: 'npm'
      - run: npm ci
      - run: npm test

  build:
    runs-on: ubuntu-latest
    needs: test
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: 'npm'
      - run: npm ci
      - run: npm run build
      - uses: actions/upload-artifact@v4
        with:
          name: build-output
          path: dist/

  deploy:
    runs-on: ubuntu-latest
    needs: build
    if: github.ref == 'refs/heads/main'
    environment: production
    steps:
      - name: Download build
        uses: actions/download-artifact@v4
        with:
          name: build-output
      - name: Deploy to production
        run: echo "Deploying..."
        env:
          DEPLOY_TOKEN: ${{ secrets.DEPLOY_TOKEN }}

Reading the Workflow

Triggers

Pipeline Best Practices

1. Keep It Fast (Under 10 Minutes)

Developers won’t wait 45 minutes for feedback. If the pipeline is slow, people stop caring about CI.

How to speed it up:

• Cache dependencies (actions/cache or built-in cache option)
• Run independent jobs in parallel
• Only run what changed (path filters)
• Use smaller, faster base images

2. Fail Fast

Run the quick checks first. If linting fails in 10 seconds, don’t waste 5 minutes running integration tests.

Lint (10s) → Unit Tests (30s) → Integration Tests (3m) → Build (2m)
  ↓ fail?       ↓ fail?             ↓ fail?
  STOP           STOP                STOP

3. Never Store Secrets in Code

Your pipeline file is in Git — anyone can read it.

4. Cache Dependencies

Don’t download node_modules or pip packages from scratch every time:

- uses: actions/setup-node@v4
  with:
    node-version: '20'
    cache: 'npm'    # ← caches node_modules automatically

5. Pin Action Versions

# ❌ Bad — can break randomly when the action updates
- uses: actions/checkout@latest

# ❌ Risky — major version can include breaking changes
- uses: actions/checkout@v4

# ✅ Best — pinned to exact commit SHA
- uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11

At minimum, pin to a major version (@v4). For security-sensitive pipelines, pin to the exact commit SHA.

6. Use Branch Protection + Required Status Checks

In GitHub Settings → Branches → Branch protection rules:

• ✅ Require status checks to pass (link your CI jobs)
• This means PRs cannot be merged unless CI passes

Deployment Strategies

When you deploy a new version, how do you do it without downtime?

Rolling Update

Instance 1: [v1] → [v2] ✅
Instance 2: [v1] → [v1] → [v2] ✅
Instance 3: [v1] → [v1] → [v1] → [v2] ✅

• Replace instances one at a time
• At any point, some run old version, some run new
• Built into Kubernetes by default
• Pro: Simple, no extra resources needed
• Con: During rollout, users hit different versions

Blue-Green

Blue (v1)  ──── [LIVE traffic] ────→ users
Green (v2) ──── [idle, testing] 

                    ↓ switch

Blue (v1)  ──── [idle]
Green (v2) ──── [LIVE traffic] ────→ users

• Two identical environments
• Deploy new version to the idle environment
• Test it. Then switch all traffic at once
• Pro: Instant rollback (switch back)
• Con: Need double the infrastructure

Canary

v1 ████████████████████ 95% of traffic
v2 █                    5% of traffic

        ↓ healthy? increase

v1 ████████████████     75% of traffic
v2 █████                25% of traffic

        ↓ healthy? increase

v2 █████████████████████ 100% of traffic

• Send a small percentage of traffic to the new version
• Monitor it. If healthy, gradually increase
• If broken, only a small percentage of users were affected
• Pro: Lowest risk, data-driven rollout
• Con: More complex to set up, need good monitoring

When to Use What

We’ll go deep into these in Phase 5. For now, know they exist and understand the tradeoffs.

Hands-On Activity

Set Up a GitHub Actions Pipeline (20 minutes)

1. Use the repo from Session 2 (or create a new one)
2. Create a simple app:

   mkdir src && echo "console.log('hello')" > src/index.js
   npm init -y

3. Add a test script to package.json:

   {
     "scripts": {
       "test": "node src/index.js",
       "lint": "echo 'lint passed'"
     }
   }

4. Create .github/workflows/ci.yml with a pipeline that:
   • Triggers on push and PR to main
   • Runs lint
   • Runs tests
   • Has a build step

Break the Pipeline, Then Fix It (10 minutes)

1. Introduce a failing test (make the script exit with code 1)
2. Push → watch the pipeline fail in GitHub Actions tab
3. Read the error logs — understand what went wrong
4. Fix the test
5. Push → watch the pipeline pass

Observe (5 minutes)

• Go to the Actions tab in GitHub
• Look at the workflow run, see the jobs and steps
• Check the timing — how long did each step take?
• Look at the logs — what information is available?

Session 3 Key Takeaways

1. CI = automated build + test on every push. Main must always be green.
2. Continuous Delivery = always deployable. Continuous Deployment = auto-deployed.
3. GitHub Actions = modern CI/CD living in your repo as YAML
4. Fail fast — lint first, unit tests next, slow tests last
5. Never put secrets in code — use GitHub Secrets or a vault
6. Deployment strategies — rolling (simple), blue-green (instant rollback), canary (lowest risk)

Discussion Questions

Think about these before next session:

1. How long does your current CI pipeline take? Could it be faster?
2. Are you doing Continuous Delivery or Continuous Deployment? What would it take to move to the next level?
3. What deployment strategy does your team use? Have you experienced a bad deployment?
4. If your pipeline fails at 3 AM, how would you know?

Next Session: Infrastructure as Code — why manual server setup is dangerous, Terraform for provisioning, Ansible for configuration, and the GitOps approach.