Adapting Airflow for Backend Infrastructure Automation

Abstract

This paper aims to present effectiveness of Airflow for management of common infrastructure management operations such as certificate renewal,operating system upgrades, configuration distribution and multi-region DR drill. Orchestration of these workflows are complex operations due to varied dependencies and a mesh of conditional workflows involving dry run validation and operation execution both in a reliable manner. Key features offered by Airflow such as DAG (Direct Acyclic Graph) based workflow execution, in-built retries, support for functional programming languages such as python allow engineers to automate day to day tasks especially for ops teams. In this paper we present example workflows tailored to infrastructure management operations.