End to End CI/CD pipeline

A CI/CD pipeline automates the software delivery process. The pipeline builds code, runs tests (CI), and safely deploys the new version of the application (CD). The goal is to minimize manual errors and deliver value to users faster and more reliably.

Here's a visual overview of the flow:

---

## 1. Source Stage: Where It All Begins 🧑‍💻

Purpose: This is the trigger for the entire pipeline. It involves managing the application's source code and collaborating with the development team.

• Core Concept: Version Control. Every change is tracked, and the repository acts as the single source of truth.

• Technologies:

  • Git: The de facto standard for distributed version control.

  • SCM Platforms: GitHub, GitLab, Bitbucket. These platforms host Git repositories and provide collaboration tools like pull/merge requests.

• How They Sync:

  • A developer pushes code changes to a specific branch (e.g., main or a feature branch) using git push.

  • This push event triggers a webhook. A webhook is an automated message (an HTTP POST payload) sent from the SCM platform (e.g., GitHub) to the CI/CD server (e.g., Jenkins). This message says, "Hey, new code is here, start the pipeline!"

• Configuration:

  • Within your GitHub/GitLab repository settings, you navigate to the "Webhooks" section.

  • You add the URL of your CI/CD server's webhook endpoint (e.g., http://<your-jenkins-server>/github-webhook/).

  • You configure this webhook to trigger on specific events, most commonly a push event.

---

## 2. Build Stage: Forging the Artifact ⚙️

Purpose: To take the raw source code and compile it into an executable or package. If the build fails, the pipeline stops, and the developer is notified immediately. This is the heart of Continuous Integration (CI).

• Core Concept: Compiling code and resolving dependencies.

• Technologies:

  • CI/CD Orchestrator: Jenkins is the classic choice. GitLab CI/CD and GitHub Actions are popular integrated solutions. Others include CircleCI and Travis CI.

  • Build Tools: Maven or Gradle (for Java), npm or Yarn (for Node.js), go build (for Golang).

• How They Sync:

  • The CI/CD orchestrator (Jenkins) receives the webhook from the Source stage.

  • It checks out the latest code from the repository using Git.

  • It then executes the build commands defined in its configuration.

• Configuration:

  • This is primarily done via Pipeline as Code. Instead of configuring jobs in a UI, you define the pipeline in a text file that is stored in the Git repository itself.

  • Jenkins: A file named Jenkinsfile (written in Groovy) defines all the stages.

    Groovy

    pipeline {
        agent any
        stages {
            stage('Build') {
                steps {
                    // For a Java project
                    sh 'mvn clean package'
                }
            }
            // Other stages follow...
        }
    }

  • GitHub Actions: A .yml file inside the .github/workflows/ directory.

---

## 3. Test Stage: Ensuring Quality 🧪

Purpose: To run a suite of automated tests to catch bugs and regressions before they reach users. This stage provides the confidence to deploy automatically.

• Core Concept: Automated quality assurance.

• Technologies:

  • Unit Testing: JUnit (Java), PyTest (Python), Jest (JavaScript). These are typically run by the build tools.

  • Static Code Analysis: SonarQube, ESLint. These tools check for code smells, potential bugs, and security vulnerabilities without executing the code.

  • Integration/API Testing: Postman (Newman), REST Assured.

• How They Sync:

  • This stage is executed by the CI/CD orchestrator (Jenkins) immediately after a successful build.

  • The orchestrator runs the test commands. It can be configured to integrate with tools like SonarQube, sending the test results and code coverage reports for analysis.

  • The pipeline will fail if the code doesn't meet the defined quality gates (e.g., unit test coverage is below 80% or there are critical security issues found by SonarQube).

• Configuration:

  • In the Jenkinsfile, you add a new stage for testing.

    Groovy

    stage('Test') {
        steps {
            // Run unit tests
            sh 'mvn test'
        }
    }
    stage('Code Analysis') {
        steps {
            // Assuming SonarQube is configured in your Jenkins instance
            withSonarQubeEnv('My-SonarQube-Server') {
                sh 'mvn sonar:sonar'
            }
        }
    }

---

## 4. Package & Store Stage: Creating the Release 📦

Purpose: To package the validated code into a versioned, immutable artifact and store it in a centralized repository.

• Core Concept: An artifact is a deployable unit. For modern microservices, this is almost always a Docker image.

• Technologies:

  • Containerization: Docker. The Dockerfile in the repository defines how to build the application image.

  • Artifact/Container Registries: JFrog Artifactory or Sonatype Nexus (for traditional artifacts like .jar, .war, .rpm). Docker Hub, Amazon ECR, Google GCR, or Azure ACR (for Docker images).

• How They Sync:

  • After the Test stage passes, the CI/CD orchestrator (Jenkins) reads the Dockerfile.

  • It runs the docker build command to create the image. The image is often tagged with a unique identifier like the Git commit hash or a build number to ensure traceability.

  • It then runs docker push to upload this newly created image to the container registry (e.g., Amazon ECR).

• Configuration:

  • You need a Dockerfile in your source code repository.

  • In the Jenkinsfile:

    Groovy

    stage('Package & Push Image') {
        steps {
            script {
                // Define image name and tag
                def imageName = "my-registry/my-app:${env.BUILD_ID}"
                // Build the Docker image
                docker.build(imageName)
                // Push the image (credentials must be configured in Jenkins)
                docker.withRegistry('https://my-registry-url', 'my-registry-credentials') {
                    docker.image(imageName).push()
                }
            }
        }
    }

---

## 5. Deploy Stage: Going Live 🚀

Purpose: To deploy the versioned artifact from the registry to a target environment (e.g., Staging, Production). This is the heart of Continuous Deployment/Delivery (CD).

• Core Concept: Automating the release process using infrastructure-as-code and configuration management.

• Technologies:

  • Container Orchestration: Kubernetes (K8s) is the industry standard.

  • Configuration Management: Ansible (agentless, push-based), Puppet (agent-based, pull-based), Chef.

  • Infrastructure as Code (IaC): Terraform, AWS CloudFormation.

• How They Sync:

  • The CI/CD orchestrator (Jenkins) triggers this final stage, often after a manual approval for production deployments (Continuous Delivery).

  • For Kubernetes: Jenkins uses kubectl (the K8s command-line tool) to apply a deployment configuration file. This file specifies which Docker image to run. The pipeline's job is to update the image tag in this file to the one it just built and pushed.

  • For Ansible: Jenkins executes an ansible-playbook command, targeting the deployment servers. The playbook contains the sequence of steps to deploy the application (e.g., pull the new Docker image, stop the old container, run the new one).

• Configuration:

  • Kubernetes: You'll have a deployment.yaml file stored in Git.

  • In the Jenkinsfile, you would use a command to update and apply this file:

    Groovy

    stage('Deploy to K8s') {
        steps {
            // 'sed' command updates the image tag in the yaml file
            sh "sed -i 's|image: my-registry/my-app:.*|image: my-registry/my-app:${env.BUILD_ID}|g' deployment.yaml"
            // Apply the updated configuration to the cluster
            // Kubeconfig must be set up in Jenkins
            sh "kubectl apply -f deployment.yaml"
        }
    }

---

## 6. Monitor & Feedback Stage: Closing the Loop 📈

Purpose: To observe the application's performance and health in real-time, collect logs, and provide feedback to the development team.

• Core Concept: Observability. You can't improve what you can't measure.

• Technologies:

  • Monitoring & Alerting: Prometheus (for collecting time-series metrics) and Grafana (for visualizing them in dashboards). Alertmanager handles alerts.

  • Log Aggregation: ELK/EFK Stack (Elasticsearch, Logstash/Fluentd, Kibana) or commercial tools like Splunk.

• How They Sync:

  • This isn't a direct pipeline stage but a continuous process. The deployed application is instrumented to expose metrics (e.g., an /metrics endpoint for Prometheus to scrape).

  • It also writes logs to stdout, which are collected by agents like Fluentd and shipped to Elasticsearch.

  • If Prometheus detects an issue (e.g., high error rate), it triggers an alert via Alertmanager, which can notify the team on Slack or PagerDuty. This information is crucial for planning the next development sprint.

• Configuration:

  • The application code includes a Prometheus client library.

  • The Kubernetes deployment configuration includes annotations that tell Prometheus to "scrape" the application's metrics endpoint.

  • Alerting rules are defined in Prometheus configuration files.

Of course. Let's ground this in a practical, end-to-end example.

Imagine we are building a simple Python web application using the Flask framework. The goal is to take this application from a developer's machine to a live deployment on a Kubernetes cluster, fully automated.

---

## Putting It All Together: A Practical Example (Python Web App)

Here's the scenario: A developer has created a simple "Hello, World!" web app. Our job is to build the CI/CD pipeline that automatically deploys it.

### The Project Files

First, let's look at the files the developer would commit to a Git repository (e.g., on GitHub).

Project Structure:

/my-python-app
├── app.py              # The Flask application code
├── requirements.txt    # Python dependencies
├── test_app.py         # The unit tests for our app
├── Dockerfile          # Instructions to build the Docker image
├── Jenkinsfile         # The CI/CD pipeline definition for Jenkins
└── k8s-deployment.yaml # The deployment manifest for Kubernetes

File Contents:

1. app.py (The Application)

   Python

   from flask import Flask
   app = Flask(__name__)
   
   @app.route('/')
   def hello_world():
       return 'Hello, DevOps World!'
   
   if __name__ == '__main__':
       app.run(host='0.0.0.0', port=5000)

2. requirements.txt (Dependencies)

   flask
   pytest

3. test_app.py (The Unit Test)

   Python

   import app
   
   def test_hello():
       client = app.app.test_client()
       response = client.get('/')
       assert response.status_code == 200
       assert b"Hello, DevOps World!" in response.data

4. Dockerfile (The Container Recipe)

   Dockerfile

   # Use an official Python runtime as a parent image
   FROM python:3.9-slim
   
   # Set the working directory in the container
   WORKDIR /app
   
   # Copy the dependencies file and install them
   COPY requirements.txt .
   RUN pip install --no-cache-dir -r requirements.txt
   
   # Copy the application code
   COPY app.py .
   
   # Expose the port the app runs on
   EXPOSE 5000
   
   # Define the command to run the application
   CMD ["python", "app.py"]

5. Jenkinsfile (The Pipeline Brains)

   Groovy

   pipeline {
       agent any
       environment {
           // Use a private Docker Hub account for the image
           // Credentials 'dockerhub-creds' must be stored in Jenkins
           DOCKER_REGISTRY = 'your-dockerhub-username'
           IMAGE_NAME = 'my-python-app'
       }
       stages {
           stage('1. Test') {
               steps {
                   echo "--- Running Unit Tests ---"
                   sh 'pip install -r requirements.txt'
                   sh 'pytest test_app.py'
               }
           }
           stage('2. Build & Push Docker Image') {
               steps {
                   script {
                       echo "--- Building & Pushing Docker Image ---"
                       // The BUILD_ID is a unique Jenkins variable
                       def imageTag = "${env.BUILD_ID}"
                       def fullImageName = "${env.DOCKER_REGISTRY}/${env.IMAGE_NAME}:${imageTag}"
   
                       // Build the image
                       docker.build(fullImageName)
   
                       // Push the image
                       docker.withRegistry("https://index.docker.io/v1/", 'dockerhub-creds') {
                           docker.image(fullImageName).push()
                       }
                   }
               }
           }
           stage('3. Deploy to Kubernetes') {
               steps {
                   echo "--- Deploying to Kubernetes Cluster ---"
                   // Credentials 'kube-config' must be stored in Jenkins
                   withKubeConfig([credentialsId: 'kube-config']) {
                       // Dynamically update the deployment file with the new image tag
                       sh "sed -i 's|image: .*|image: ${env.DOCKER_REGISTRY}/${env.IMAGE_NAME}:${env.BUILD_ID}|g' k8s-deployment.yaml"
   
                       // Apply the updated configuration
                       sh "kubectl apply -f k8s-deployment.yaml"
                   }
               }
           }
       }
   }

6. k8s-deployment.yaml (The Deployment Instructions)

   YAML

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: my-python-app-deployment
   spec:
     replicas: 2
     selector:
       matchLabels:
         app: my-python-app
     template:
       metadata:
         labels:
           app: my-python-app
       spec:
         containers:
         - name: python-app-container
           # This 'image' line is a placeholder that Jenkins will update
           image: your-dockerhub-username/my-python-app:latest
           ports:
           - containerPort: 5000

---

### The Automated Flow in Action

1. Source 🧑‍💻: The developer makes a change to app.py and runs git push origin main.

2. Trigger: GitHub receives the push and immediately sends a webhook to the pre-configured Jenkins server.

3. Initiate: Jenkins receives the webhook, finds the Jenkinsfile in the main branch of the repository, and starts a new pipeline build (e.g., Build #42).

4. Stage 1: Test 🧪:

   • Jenkins executes the Test stage.

   • It runs pip install -r requirements.txt to install Flask and Pytest.

   • It then runs pytest test_app.py. The test passes. The pipeline proceeds.

5. Stage 2: Package & Store 📦:

   • Jenkins executes the Build & Push Docker Image stage.

   • It builds the Docker image using the Dockerfile. Let's say the build ID is 42. The resulting image will be named your-dockerhub-username/my-python-app:42.

   • Using the stored credentials, Jenkins logs into Docker Hub and pushes this new image to the registry. The artifact is now stored and versioned.

6. Stage 3: Deploy 🚀:

   • Jenkins moves to the final Deploy to Kubernetes stage.

   • It uses the sed command to find the image: line in k8s-deployment.yaml and replaces it with the exact image it just pushed: image: your-dockerhub-username/my-python-app:42.

   • Using the Kubernetes config credentials, it runs kubectl apply -f k8s-deployment.yaml.

   • Kubernetes receives this instruction. It sees the deployment needs an updated image. It performs a rolling update: it creates new pods with the new ...:42 image and, once they are healthy, terminates the old pods. This ensures zero downtime.

7. Operate & Monitor 📈: The new application pods are now running. Prometheus scrapes their metrics, and Fluentd collects their logs. If the error rate suddenly spikes, the on-call team gets an alert.

Within minutes of the git push, the developer's change is tested, packaged, and live in production, all without any manual intervention. This is the power of a fully integrated CI/CD pipeline.