Portfolio

DevOps Gitlab CI/CD + Ansible Molecule Pipeline Design

DevOps GitLab CI/CD + Ansible Molecule Pipeline

8 February 2024 / Comments

Introduction

Streamlining the development, testing, and deployment of infrastructure and applications is a cornerstone of modern DevOps practices. GitLab CI/CD provides a robust platform for automating pipelines, and Ansible Molecule offers a powerful framework for testing Ansible roles prior to deployment. In this article, I'll demonstrate how to marry these tools to create a cohesive GitLab DevOps pipeline, including integration testing and provisioning.

You can find the source code for this project in my GitLab repository here: Testing Ansible with Molecule in Docker

Key Components

  • GitLab CI/CD: The central orchestration platform for our automated build, test, and deploy processes.
  • Ansible: The configuration management and IT automation engine, used to define our infrastructure as code.
  • Ansible Roles: Modular and reusable Ansible code blocks to structure infrastructure configuration.
  • Molecule: A framework specifically designed for testing the logic and execution of Ansible roles.

The Pipeline

Our GitLab DevOps pipeline will consist of three primary stages:

  1. Container Build:
    • A Dockerfile defines a container image containing Ansible, Molecule, and necessary dependencies.
    • GitLab CI/CD automates the image build and pushes it to a container registry (GitLab Container Registry).
  2. Ansible Role Testing with Molecule:
    • Molecule test scenarios are defined, ensuring our Ansible roles function as intended in the various architectures and OS.
    • Tests are executed within the built container, providing isolation and consistency.
  3. Provisioning:
    • An Ansible playbook utilizes the tested roles to configure target infrastructure.
    • GitLab CI/CD executes the playbook against designated environments (dev, staging, production).

Code example (.gitlab-ci.yml)

Build multi-arch Docker images rootless with Kaninko

Defined in .gitlab-ci.yml the CI the stages:

  1. The GitLab Runners build the containers, for 2 architectures, arm64 and amd64, in separated instances.
  2. Create Docker manifest for the 2 architectures and push the containers to GitLab Registry

Understanding the Structure in depth

  • Stages: The pipeline is divided into two stages
    • build: This is where the Docker image is built for different architectures.
    • multiarch-manifest: This creates a multi-architecture manifest to seamlessly support various architectures.
  • Jobs: We have two build jobs (kaniko-amd64, kaniko-arm64) and one manifest job (multiarch-manifest).
1. kaniko-amd64 & kaniko-arm64 Jobs
  • Purpose: Build Docker images for AMD64 (kaniko-amd64) and ARM64 (kaniko-arm64) architectures.
  • Image: Use the Kaniko executor image (gcr.io/kaniko-project/executor:debug) to build Docker images without Docker-in-Docker.
  • Variables
    • KANIKO_ARGS: Additional flags for the Kaniko executor.
    • KANIKO_BUILD_CONTEXT: The directory where the Dockerfile is located.
  • Runner Tags: associate the workload to specific runners
    • kaniko-arm64 is specifically tagged to run on an ARM64, in this case a private runner.
    • kaniko-amd64 is tagged to run on an AMD64 runner.
kaniko-amd64:
  variables:
    # Additional options for Kaniko executor.
    # For more details see https://github.com/GoogleContainerTools/kaniko/blob/master/README.md#additional-flags
    KANIKO_ARGS: ""
    KANIKO_BUILD_CONTEXT: $CI_PROJECT_DIR
  stage: build
  image:
    # For latest releases see https://github.com/GoogleContainerTools/kaniko/releases
    # Only debug/*-debug versions of the Kaniko image are known to work within Gitlab CI
    name: gcr.io/kaniko-project/executor:debug
    entrypoint: [""]
  script:
    # if the user provide IMAGE_TAG then use it, else build the image tag using the default logic.
    # Default logic
    # Compose docker tag name
    # Git Branch/Tag to Docker Image Tag Mapping
    #   * Default Branch: main -> latest
    #   * Branch: feature/my-feature -> branch-feature-my-feature
    #   * Tag: v1.0.0/beta2 -> v1.0.0-beta2
    - |
      if [ -z ${IMAGE_TAG+x} ]; then
        if [ "$CI_COMMIT_REF_NAME" = $CI_DEFAULT_BRANCH ]; then
            VERSION="latest"
          elif [ -n "$CI_COMMIT_TAG" ];then
            NOSLASH=$(echo "$CI_COMMIT_TAG" | tr -s / - )
            SANITIZED="${NOSLASH//[^a-zA-Z0-9.-]/}"
            VERSION="$SANITIZED"
          else \
            NOSLASH=$(echo "$CI_COMMIT_REF_NAME" | tr -s / - )
            SANITIZED="${NOSLASH//[^a-zA-Z0-9-]/}"
            VERSION="branch-$SANITIZED"
          fi
        export IMAGE_TAG=$CI_REGISTRY_IMAGE:$VERSION-amd64
      fi
    - echo $IMAGE_TAG
    - mkdir -p /kaniko/.docker
    # Write credentials to access Gitlab Container Registry within the runner/ci
    - echo "{\"auths\":{\"$CI_REGISTRY\":{\"auth\":\"$(echo -n ${CI_REGISTRY_USER}:${CI_REGISTRY_PASSWORD} | base64 | tr -d '\n')\"}}}" > /kaniko/.docker/config.json
    # Build and push the container. To disable push add --no-push
    - DOCKERFILE_PATH=${DOCKERFILE_PATH:-"$KANIKO_BUILD_CONTEXT/Dockerfile"}
    - /kaniko/executor --context $KANIKO_BUILD_CONTEXT --dockerfile $DOCKERFILE_PATH --destination $IMAGE_TAG $KANIKO_ARGS
  # Run this job in a branch/tag where a Dockerfile exists
  rules:
    - exists:
        - Dockerfile
    # only tags
    - if: "$CI_COMMIT_TAG"
    # custom Dockerfile path
    - if: $DOCKERFILE_PATH
    # custom build context without an explicit Dockerfile path
    - if: $KANIKO_BUILD_CONTEXT != $CI_PROJECT_DIR
2. multiarch-manifest Job
  • Purpose: Creates a multi-architecture manifest, enabling Docker to pull the appropriate image based on the user's system architecture.
  • Image: Uses a standard docker:latest image, as this task relies on the Docker CLI.
  • Services:
    • docker:dind: Leverages Docker-in-Docker to provide Docker containers inside the runner.
stage: multiarch-manifest
  image:
    # For latest releases see https://github.com/GoogleContainerTools/kaniko/releases
    # Only debug/*-debug versions of the Kaniko image are known to work within Gitlab CI
    name: docker:latest
    entrypoint: [""]
  services:
    - docker:dind
  before_script:
    # Connect to GitLab Docker in Docker DIND
    - mkdir -pv ~/.docker
    - cp -v $DOCKER_TLS_CERTDIR/client/* ~/.docker   # Copy default docker certs
    - docker info   # Show connection to docker daemon
    # Login... if necessary
    - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
  script:
    # The Docker script use another tag format `:7-8-5`, use Kaniko style for coerence
    - |
      if [ -z ${IMAGE_TAG+x} ]; then
        if [ "$CI_COMMIT_REF_NAME" = $CI_DEFAULT_BRANCH ]; then
            VERSION="latest"
          elif [ -n "$CI_COMMIT_TAG" ];then
            NOSLASH=$(echo "$CI_COMMIT_TAG" | tr -s / - )
            SANITIZED="${NOSLASH//[^a-zA-Z0-9.-]/}"
            VERSION="$SANITIZED"
          else \
            NOSLASH=$(echo "$CI_COMMIT_REF_NAME" | tr -s / - )
            SANITIZED="${NOSLASH//[^a-zA-Z0-9-]/}"
            VERSION="branch-$SANITIZED"
          fi
        export IMAGE_TAG=$CI_REGISTRY_IMAGE:$VERSION
      fi
    # Example Kaniko tag: registry.gitlab.com/aleliga/docker-ansible-runner:7.8.5
    # Pull images
    - docker pull $IMAGE_TAG-arm64
    - docker pull $IMAGE_TAG-amd64
    # Fill variables for `docker manifest`
    - export MANIFEST_LIST=$IMAGE_TAG && export MANIFEST_BASE=$MANIFEST_LIST
    - echo "Debug vars $MANIFEST_LIST $MANIFEST_BASE"
    # Create docker manifest
    - echo "docker manifest create $MANIFEST_LIST $MANIFEST_BASE-arm64 $MANIFEST_BASE-amd64"
    - docker manifest create $MANIFEST_LIST $MANIFEST_BASE-arm64 $MANIFEST_BASE-amd64
    # Annotate arch
    - echo "docker manifest annotate --os linux --arch arm64 $MANIFEST_LIST $MANIFEST_BASE-arm64"
    - docker manifest annotate --os linux --arch arm64 $MANIFEST_LIST $MANIFEST_BASE-arm64
    - echo "docker manifest annotate --os linux --arch amd64 $MANIFEST_LIST $MANIFEST_BASE-amd64"
    - docker manifest annotate --os linux --arch amd64 $MANIFEST_LIST $MANIFEST_BASE-amd64
    # Push manifest
    - echo "docker manifest push $MANIFEST_LIST"
    - docker manifest push $MANIFEST_LIST
  rules:
    - exists:
        - Dockerfile
    # only tags
    - if: "$CI_COMMIT_TAG"
    # custom Dockerfile path
    - if: $DOCKERFILE_PATH
    # custom build context without an explicit Dockerfile path
    - if: $KANIKO_BUILD_CONTEXT != $CI_PROJECT_DIR

Run Ansible Molecule in GitLab CI/CD

A secondary CI pipeline, is dedicated to the integrations tests of Ansible Playbooks and Roles with molecule, using the previous built container

stages:
  - molecule-test
variables:
CONTAINER_RELEASE_IMAGE: "$CI_REGISTRY_IMAGE:$CI_COMMIT_TAG"
SHELL: /bin/bash
# Registry login info
CI_REGISTRY: registry.gitlab.com
before_script:
  - echo "Before script"
  - mkdir -p ~/.docker/
  - echo "{\"auths\":{\"$CI_REGISTRY\":{\"auth\":\"$(echo -n ${CI_REGISTRY_USER}:${CI_REGISTRY_PASSWORD} | base64)\"}}}" >> ~/.docker/config.json
molecule:
image: registry.gitlab.com/aleliga/docker-ansible-molecule:latest
stage: molecule-test
services:
    - docker:dind
script:
  - echo "Run molecule destroy"
  - "$CI_PROJECT_DIR/bin/molecule-ci-direct.sh destroy"
  - echo "Remove temp files"
  - rm -fr ~/.cache/molecule/
  - echo "Run molecule test"
  - "$CI_PROJECT_DIR/bin/molecule-ci-direct.sh test"

Deploy with Ansible in CD

The last pipeline, deploy the Ansible playbooks from GitLab Runner, using DinD (Docker in Docker)

variables:
  # Deploy repository
  PRIVATE_REPO="gitlab.com/USER/PRIVATE_REPO.git"
  
  # Docker DIND vars
  DOCKER_HOST: "tcp://docker:2376"
  DOCKER_TLS_CERTDIR: "/certs"
  DOCKER_TLS_VERIFY: 1
ansible-deploy:
  image: registry.gitlab.com/aleliga/docker-ansible-molecule:latest
  stage: deploy
  services:
    - docker:dind
  before_script:
    - 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
    - eval $(ssh-agent -s)
    - echo -e $CI_SSH_PRIVATE_KEY
    - echo -e "$CI_SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add - > /dev/null
    - mkdir -p ~/.ssh
    - chmod 700 ~/.ssh
    - cd ~/
    - mkdir -p $PERSONAL_REPO_DIR
    - apt-get update && apt-get install git -qqy
  script:
    - git clone https://"$ANSIBLE_USER":"$ANSIBLE_DEPLOY_TOKEN"@"$PRIVATE_REPO" $PERSONAL_REPO_DIR
  # Run Ansible "upgrade-hosts" to deploy in production
    - cd $PERSONAL_REPO_DIR
    - echo "Deploy to production server"
    - bin/gitlabci-upgrade-hosts.sh

Benefits

  • Enhanced Reliability: Automated testing with Molecule increases confidence in Ansible roles, preventing errors from slipping into production environments.
  • Faster Iteration: A well-defined CI/CD pipeline enables faster development and changes with a higher degree of safety.
  • Infrastructure as Code: Ansible ensures predictable and repeatable deployments across environments.

Considerations

  • Secure Variable Handling: Use GitLab CI/CD Secret Variables and Vault solutions for storing sensitive data like credentials.
  • Environment Management: Consider separating pipeline configurations for development, staging, and production environments.

Conclusion

Integrating GitLab CI/CD and Ansible Molecule empowers a solid DevOps workflow. It ensures reliable infrastructure changes, accelerates delivery time, and reduces the risk of deployment errors.

Feel free to explore the repository, contribute, or use it as a reference. Let me know if you have any feedback or suggestions!

DevOps
Linux