Pipeline Architecture¶

Every project repo contains a Jenkinsfile at its root. Jenkins discovers these automatically via the GitHub Organization Folder job and runs them on every push and pull request.

Where things live¶

magi/                            ← you are here
├── casc.yml                     ← Docker agent templates, credentials, shared library registration
├── Dockerfile                   ← Jenkins controller image with all plugins baked in
└── docker-compose.yml           ← Runs the controller + mounts Docker socket

melchior/                        ← separate repo
└── vars/
    ├── detectAgent.groovy       ← infers agent label from project files
    ├── runTests.groovy          ← shared Test + Archive stage logic
    ├── deployMkdocs.groovy      ← shared Deploy Docs stage logic
    └── generateChangelog.groovy ← shared Update Changelog stage logic

<any-project-repo>/
├── Jenkinsfile                  ← calls shared library steps; defines what runs
├── cliff.toml                   ← git-cliff config for changelog generation
└── src/ tests/ etc.

Build agent¶

Each build runs inside a fresh Docker container spun up on demand and discarded after the build. The Jenkins controller never runs build code — it only orchestrates. The Docker socket mount (/var/run/docker.sock) allows the controller to launch agent containers via Docker Desktop.

Agent selection¶

Three ways a project declares its build environment, in order of preference:

Scenario	Agent declaration	`casc.yml` change?
Single language, standard runtime	`agent { label 'python-3.14' }`	Never
Multi-language, different stages	`agent none` + per-stage labels	Never
Multi-language, same stage	`agent { dockerfile { filename 'Dockerfile.ci' } }`	Never
New commonly-used language	Add template to `casc.yml`	Yes — one time

Pre-defined labels (configured in casc.yml): python-3.14, node-20, java-21, go-1.22, dotnet-8, ruby-3.3

Dockerfile.ci — for projects that need runtimes not covered by a single label, or need a custom combination. Placed in the project repo root, built and cached by Docker on first run:

# Python backend + Node frontend in one image
FROM python:3.14
RUN curl -fsSL https://deb.nodesource.com/setup_20.x | bash - \
    && apt-get install -y nodejs

Auto-detection via shared library¶

When using the shared library, detectAgent() selects the agent automatically based on files it finds in the workspace — no agent declaration needed in the Jenkinsfile:

File present	Agent used
`Dockerfile.ci`	`dockerfile { filename 'Dockerfile.ci' }` — highest priority
`pyproject.toml` / `setup.py`	`label 'python-3.14'`
`package.json`	`label 'node-20'`
`pom.xml` / `build.gradle`	`label 'java-21'`
`go.mod`	`label 'go-1.22'`
`.csproj` / `.sln`	`label 'dotnet-8'`
`Gemfile`	`label 'ruby-3.3'`

Stages¶

Install¶

Lives in: project Jenkinsfile (or shared step) What it does: Installs the project and its dev dependencies into the ephemeral agent.

Test¶

Lives in: project Jenkinsfile (or runTests shared step) What it does: Runs the test suite, captures results. Failure handling: catchError(buildResult: 'UNSTABLE') — build continues even if tests fail so artifacts are still archived. Post-stage: JUnit plugin publishes test-results.xml as a trend graph in Jenkins.

Key environment variables injected:

Variable	Value	Purpose
`CI`	`true`	Suppresses local archive creation
`VERSION`	`v${BUILD_NUMBER}`	Stamped into test output and result files

Archive Artifacts¶

Lives in: project Jenkinsfile (or runTests shared step) Condition: only runs if artifact directory exists What it does: Archives test output and result files into Jenkins build storage. Retention: 30 days (configured via "Discard Old Builds" on the job).

Update Changelog¶

Lives in: project Jenkinsfile (or generateChangelog shared step) Condition: main branch only What it does:

Runs git-cliff via Docker to regenerate CHANGELOG.md from Conventional Commits history
Commits CHANGELOG.md back to main with [skip ci] to prevent a build loop
Skips the commit if CHANGELOG.md has no changes

cliff.toml in each project repo controls changelog format and filters noise commits.

Deploy Docs¶

Lives in: project Jenkinsfile (or deployMkdocs shared step) Condition: main branch only What it does:

Installs doc dependencies
Rewrites the remote URL with a Jenkins CI App token via withCredentials([gitHubApp(...)])
Runs mkdocs gh-deploy --force to push built docs to the gh-pages branch

Full Jenkinsfile (before shared library)¶

pipeline {
    agent {
        docker { image 'python:3.14' }
    }
    environment {
        TMP_FILES_DIR = 'tmp-test-files'
        VERSION       = "v${BUILD_NUMBER}"
        CI            = 'true'
    }
    stages {
        stage('Install') {
            steps {
                sh 'pip install -e ".[dev]"'
            }
        }
        stage('Test') {
            steps {
                catchError(buildResult: 'UNSTABLE', stageResult: 'UNSTABLE') {
                    sh 'pytest'
                }
            }
            post {
                always { junit 'test-results.xml' }
            }
        }
        stage('Archive Artifacts') {
            when {
                expression { fileExists('tmp-test-files') }
            }
            steps {
                archiveArtifacts artifacts: 'tmp-test-files/**, test-output.txt, test-results.xml',
                                 fingerprint: true
            }
        }
        stage('Update Changelog') {
            when { branch 'main' }
            steps {
                sh 'docker run --rm -v $(pwd):/app orhunp/git-cliff:latest --output CHANGELOG.md'
                withCredentials([gitHubApp(credentialsId: 'jenkins-ci-app', variable: 'GH_TOKEN')]) {
                    sh '''
                        git config user.email "jenkins@example.com"
                        git config user.name "Jenkins"
                        git remote set-url origin https://x-access-token:${GH_TOKEN}@github.com/<your-github-username>/<repo>.git
                        if ! git diff --quiet CHANGELOG.md; then
                            git add CHANGELOG.md
                            git commit -m "chore: update changelog [skip ci]"
                            git push origin main
                        fi
                    '''
                }
            }
        }
        stage('Deploy Docs') {
            when { branch 'main' }
            steps {
                sh 'pip install -e ".[docs]"'
                withCredentials([gitHubApp(credentialsId: 'jenkins-ci-app', variable: 'GH_TOKEN')]) {
                    sh '''
                        git config user.email "jenkins@example.com"
                        git config user.name "Jenkins"
                        git remote set-url origin https://x-access-token:${GH_TOKEN}@github.com/<your-github-username>/<repo>.git
                        mkdocs gh-deploy --force
                    '''
                }
            }
        }
    }
    post {
        always { cleanWs() }
    }
}

Thin Jenkinsfile (after shared library)¶

@Library('shared') _

standardPipeline([
    changelog: true,
    deployDocs: true,
])

Or with explicit stages:

@Library('shared') _

pipeline {
    agent { script { detectAgent() } }
    stages {
        stage('Test')             { steps { runTests() } }
        stage('Update Changelog') { when { branch 'main' }
                                    steps { generateChangelog() } }
        stage('Deploy Docs')      { when { branch 'main' }
                                    steps { deployMkdocs() } }
    }
}

Trigger flow¶

push / PR to GitHub
      ↓
Jenkins CI App sends webhook → JENKINS_URL/github-webhook/
      ↓
GitHub Branch Source plugin receives it
      ↓
matching job triggered (branch or PR build)
      ↓
agent container spun up from casc.yml template
      ↓
stages run → results posted back to GitHub commit status
      ↓
agent container destroyed, workspace cleaned