Rollback for microservices with Ansible and Jenkins


It is required to generate an input with dropdown.

stage('Select rollback date') { steps { script { def userInput = false try { timeout(time: 120, unit: 'SECONDS') { userInput = input(id: 'userInput', message: 'Select a date to rollback', parameters: [ choice(name: 'rollback_date', choices: gather_rollback_dates(), description: 'One or more services have rollback at this date')]) } } catch(err) { } if (userInput) { print('Performing rollback') env.

DATE = userInput } else { print('Skip rollback') } } } }It looks like this:Perform a rollbackWe have 5 actions for a rollback:Read json from previous stepFind missing images for the selected dateGet marathon service ids from docker idsChange marathon app’s configUpdate app in marathonAnsible sideNothing special here.

Just call python.

– name: "Perform rollbacks" shell: "source activate /tmp/{{ role_name }}/{{ conda_env }} ; {{ item }}" with_items: – pip install -r /tmp/{{ role_name }}/requirements.

txt – "python /tmp/{{ role_name }}/rollback.

py run –date={{ date }} –env={{ env }} –slack={{ slack }} –user={{ dcos_user }} –dump={{ dump_path }} –pwd={{ dcos_password }}" tags: – runPython sideLet’s start with run methodRead json and select all available images for a selected date.

def run(date, env, slack, user, pwd, dump): json_data = read_rollbacks(dump) all_rollbacks = OrderedDict(sorted(json_data['rollbacks'].

items(), key=lambda x: x[0])) repos = json_data['all'] images = all_rollbacks[date]If images for some repos are missing — we need to find their older versions.

Add this to your run method:if len(repos) > 1 and len(repos) > len(images): get_missing_images(date, repos, all_rollbacks)Where get_missing_images just goes through all_rollbacks and selects image with nearest date for each missing image.

def get_missing_images(date, repos, all_rollbacks): images = all_rollbacks[date] # select available images found_services = [list(rb.

values())[0] for rb in images] # get services from images missing = list(set(repos) – set(found_services)) # substract to get missing for service in missing: # populate images with rollback for every missing rollback = get_nearest_date(service, date, all_rollbacks) if rollback is None: print(f"Previous rollback for {service} not found") else: images += [rollback] def get_nearest_date(repo, date, all_rollbacks): for d, images in reversed(all_rollbacks.

items()): if d < date: for rollback, image in images[0].

items(): if image == repo: return {rollback: image} return NoneAfter we have our images populated we need to get marathon service ids.

Our marathon ids uses standard /<department>/<environment>/<project>/<service-name>.

At this step we have only service-name, so we should create a binding to Maration id.

We can do it by listing all applications running in Maration and filtering them by the environment and service name (I haven’t found better solution).

def get_service_ids(env: str, images: list, user: str, pwd: str) -> dict: ids_only = get_marathon_ids_for_env(env, user, pwd) # all running services for env services = {} for rollback in images: tag = list(rollback.

keys())[0] id_part = rollback[tag] real_id = list(filter(lambda x: x.

endswith(id_part), ids_only)) # filter by service-name if not real_id: raise Exception(f"Id {id_part} not found") services[real_id[0]] = tag return services def get_marathon_ids_for_env(env: str, user: str, pwd: str): res = call_with_output(f'dcos auth login –username={user} –password={pwd}') if res.


strip() != 'Login successful!': raise Exception("Can't login to dcos cli") all_services = call_with_output('dcos marathon app list') matched = list(filter(lambda x: x.

startswith(f"/ds/{env}"), all_services.


split('.'))) return [m.

split(' ')[0] for m in matched]After we have service ids we can iterate through them and do a rollback for each.

Add this to your run method:services = get_service_ids(env, images, user, pwd) for service_id, service_tag in services.

items(): if slack is not None: notify_slack(slack, f"Rollback { service_id }: { service_tag }") print(do_deploy(service_id, service_tag))Well, that’s all.

Don’t forget to add slack notifications for the rollback.

Jenkins sidePython part was the most complex.

On Jenkins side you just need to call Ansible with run tag and selected date.

stage('Rollback') { when { expression { return env.

DATE != null } } steps { ansiblePlaybook( playbook: "${env.


yaml", inventory: "inventories/dev/hosts.

ini", credentialsId: <your git user credentials id>', extras: '-e "date=' + "${env.

DATE}" + ' env=' + "${params.

environment}" + ' slack=' + "${env.

SLACK_CALLBACK}" + ' dump_path=' + "/tmp" + '" -v', tags: "run") }}Summing upCurrent solution is quite complex, but it allows you to run rollbacks both from Ansible via cli and from Jenkins.

The second one is preferred, as you can see the user who approved the rollback.

Have a nice firefighting and hope you’ll never have a need in rollbacks!.

. More details

Leave a Reply