Continuous Delivery and Rolling Upgrades with Ansible

While working on a application running on premises, I noticed that Ansible comes in handy to manage product upgrade or automating the release pipeline. My starting point is Rest application implemented in JVM/Scala and running in 15 nodes behind a HAProxy which provides high availability and load balancing. To be honest the deployment involves two data center and a nginx proxy to balance the load between two data center, but it’s not really relevant for the purpose of ansible use.

Background

The application is distributed as a binary zip tar ball, created by Sbt native packager, and uploaded after every release to an artifactory repository. The application package is a linux package binary application. Basically a zip with all dependencies and an a runnable script. If one might question why not to use a fat jar instead you might want to read the-fault-in-our-jars-why-we-stopped-building-fat-jars. I had the same issue that in the post, my third parties dependencies overlap META-INF/services and for me it’s was a risk aggregating all dependencies jar in a fat jar.

The problem

Rolling up new software updates across 15 nodes in production with zero down time. My nodes are bare metal instances with static ips, so for me ansible comes in handy to handle the deployment. In short we are going to use Ansible to automate follow steps:

  1. Download binary application from artifactory.

  2. Extract binary application to application home directory.

  3. Update the soft link pointing to the new folder.

  4. Restart systemd service.

We will create a ansible directory layout with role and tasks to manage new roll outs.

Let’s start by showcasing the folder structure that we should aim for.

├── group_vars
│     └── all.yml
├──  roles
│     └── rest-application
│         ├── defaults
│         │   └── main.yml
│         ├── tasks
│         └── templates
│              └── application.ini.j2
├──  production.yml
├──  staging.yml
└──  rest-application.yml

In the folder structure above:

  • production.yml ansible inventory for production nodes

  • group_var/all.yml place holder file for common variables

  • roles/rest-application ansible role which will handle the software upgrade

  • rest-application.yml ansible playbook which binds them all.

Ansible inventory files

Ansible inventory represents a group of hosts that you want to automate tasks on. In my case a group of hosts tagged as “production environment” where my application will run. Ansible supports inventory files in ini format or YAML format. I see more convenient and readable to use YML format.

The inventory production file looks like:

---
all:
  vars:
    version: 2023.01
    # Comment out this line if artifact is not SNAPSHOT
    version_suffix: "-SNAPSHOT"

    serial: 2
    can_access_artifactory: false

    ansible_python_interpreter: /opt/python-2.7.10/bin/python
    service_stop_command: "systemctl stop rest-application.service"
    service_start_command: "systemctl start rest-application.service"
    rest_application_jvm_opts: |
      -J-Xmx1G
      -J-XX:+UseG1GC
      -J-XX:MaxGCPauseMillis=1000

  children:
    rest_application:
      hosts:
        node1.production.net:
        node2.production.net:
          ansible_python_interpreter: '/usr/bin/python3'
          application_java_home: "/java_11/jdk-11.0.11"
        node3.production.net:

As one might notice the YML format allow to define and override ansible variables for all environment or per host.

At this point, if you have shared your ssh public key in target hosts, you could run a ping playbook

$ ansible -i production.yml -m ping all

node1.production.net | SUCCESS => {
  "changed": false,
  "ping": "pong"
}

Ansible Roles

Using ansible roles is kind of easy to automate the steps described above.

---
# tasks file: main.yml 

- set_fact:
    artifactory_url: "{{ artifactory_release_repository }}"    

- set_fact:
    artifactory_url: "{{ artifactory_snapshot_repository }}"
  when:
    is_snapshot

- name: Download the artifact from artifactory
  get_url:
    url: "{{ rest_application_artifact_url }}"
    dest: "/tmp/{{ rest_application_artifact_id }}-{{ version }}.tgz"
  when:
    can_access_artifactory
  tags: [ download-artifact ]

- name: Download the artifact to local
  delegate_to: localhost
  get_url:
    url: "{{ rest_application_artifact_url }}"
    dest: "/tmp/{{ rest_application_artifact_id }}-{{ version }}.tgz"
  when:
    not can_access_artifactory
  tags: [ download-artifact ]

- name: Copy the artifact to target
  copy:
    src: "/tmp/{{ rest_application_artifact_id }}-{{ version }}.tgz"
    dest: "/tmp/{{ rest_application_artifact_id }}-{{ version }}.tgz"
    mode: u=rw,g=rw,o=r
  when:
    not can_access_artifactory
  tags: [ download-artifact ]

- name: Extract artifact to release directory
  unarchive:
    copy: no
    src: "/tmp/{{ rest_application_artifact_id }}-{{ version }}.tgz"
    dest: "{{ rest_application_releases_dir }}"
    mode: u=rwx,g=rwx+s,o=rx

- name: Setting rw permissions to group
  file:
    dest: "{{ rest_application_install_dir }}"
    mode: "u=rwX,g=rwX,o=rX"
    recurse: "yes"

- name: Make the link to application launcher
  file:
    src: "{{ rest_application_install_dir }}/bin/{{ rest_application_artifact_id }}"
    dest: "{{ rest_application_install_dir }}/{{ rest_application_artifact_id }}"
    state: link

- name: Configure JVM settings
  template:
    src: "application.ini.j2"
    dest: "{{ rest_application_install_dir }}/conf/application.ini"

- name: Stop service
  shell: "{{ rest_application_service_stop_command }}"
  register: log_result
  ignore_errors: yes
  tags: [ restart-service ]

- name: Update current release link
  file:
    src: "{{ rest_application_install_dir }}"
    dest: "/apps/rest-application/current"
    state: link
  tags: [ update-current-link ]

- name: Start service
  shell: "{{ rest_application_service_start_command }}"
  register: log_result
  failed_when: "log_result.rc not in [ 0 ]"
  tags: [ restart-service ]

- name: Show start command output
  debug: msg="{{ log_result.stdout_lines }}"
  when: log_result
  tags: [ restart-service ]

- name: Wait service to be healthy
  uri:
    url: "{{ rest_application_health_check_url }}"
    status_code: 200
  register: result
  until: result.status == 200
  retries: 5
  delay: 10
  tags: [ health-check ]

- name: Remove downloaded artifact
  file:
    path: "/tmp/{{ rest_application_artifact_id }}-{{ version }}.tgz"
    state: absent

- name: Show install dir
  debug: msg="{{ rest_application_install_dir }}"

Ansible playbook

---

# play file: rest-application.yml

- hosts:
    rest_application
  roles:
    - rest-application

Running the playbook

When the ansible layout is ready you can run the playbook which just created

$ ansible-playbook -i production.yml rest-application.yml