Challenge
Gene editing is a revolutionary and complex field that has the potential to cure a range of illnesses, from Huntington’s to sickle cell anemia. The technology used to manage the processes required to harness this potential is just as complex, which can be a headache for software developers.
Continuous maintenance of a custom genome engineering platform was preventing a national biotechnology firm from focusing on software solutions that could push the company forward. The company reached out to Six Feet Up to implement a scheduler for its manufacturing process.
Specifically, Six Feet Up helped the company:
- dissect the company’s custom workflow management software,
- implement Apache Airflow into the company's internal code pipeline; and
- migrate existing workflows from the custom software to Airflow.
The company requested that existing workflows be migrated to Airflow because it is easy to use and has ample documentation to aid in onboarding new developers.
Implementation Details
The company's custom web framework uses Python and SQLAlchemy to manage workflows for a variety of processes, including order fulfillment and lab work. Each of these workflows includes a variety of tasks and timelines. The framework was developed and built on over time, and, as a result, it became very complex and featured a significant amount of cruft.
Before Airflow could be implemented, Six Feet Up’s experts had to gain a deeper understanding of the custom framework to ensure a smooth transition.
Understanding a Custom Framework
First, Six Feet Up’s developers used Docker to create a local Airflow development environment. From there, they worked with the company's developers to answer initial questions before diving into the code base to figure out the inner workings of the software. This process included:
- finding the entry point into the code,
- tracing the code from that entry point to understand the workflow engine; and
- determining how individual workflows interacted with the workflow engine and how code was structured for each task.
Implementing Airflow
Six Feet Up's developers started by modifying the Airflow quickstart (generic steps used to implement Airflow) so that Airflow could access the company's package repository. This also allowed Airflow to be integrated into the company's data pipeline with very little setup work.
Then, Six Feet Up’s experts established a local development environment that linked Airflow to the company's Django-based backend services. Developers modified the integration tests to trigger DAGs in Airflow, and when the workflow migrations to Airflow were completed, modifications were made to those tests to confirm the data was accurate on completion of the DAG runs.
Migrating Workflows
With the groundwork done, Six Feet Up’s developers could begin the work of migrating workflows to Airflow. The company's workflows are organized into conductors, or umbrella workflows, that manage sub-workflows.
While the goal was for the final structure of the conductors and workflows to roughly match the existing structure, Six Feet Up’s experts consulted with the company's developers about condensing some of the workflows to make them run more quickly. This process required collaboration because while parallelizing tasks could boost efficiency, it could also make the workflow structure more complex, which would make onboarding developers more difficult.
Six Feet Up’s experts translated several workflows to Airflow, then used the company's user interface to run through the workflows. The company's Django-based backend services would then process the data coming through the workflow engine in Airflow. As a result, the Six Feet Up team could match the data with what would happen when that process was run through the company's custom web framework.
Results
Six Feet Up’s experts created an actionable roadmap for the conversion of future workflows to Airflow and developed a procedure to easily create workflows in Airflow by mixing and matching code representing units of work.
Once all of the workflows are migrated to Airflow in the development environment, the Airflow workflows will be deployed into production and used to successfully schedule the company's manufacturing process.
