The best way you retrieve variables from Airflow can affect the efficiency of your DAGs

What occurs if a number of information pipelines must work together with the identical API endpoint? Would you actually should declare this endpoint in each pipeline? In case this endpoint adjustments within the close to future, you’ll have to replace its worth in each single file.

Airflow variables are easy but priceless constructs, used to stop redundant declarations throughout a number of DAGs. They’re merely objects consisting of a key and a JSON serializable worth, saved in Airflow’s metadata database.

And what in case your code makes use of tokens or different sort of secrets and techniques? Hardcoding them in plain-text doesn’t appear to be a safe strategy. Past decreasing repetition, Airflow variables additionally assist in managing delicate data. With six other ways to outline variables in Airflow, choosing the suitable methodology is essential for guaranteeing safety and portability.

An usually neglected side is the affect that variable retrieval has on Airflow efficiency. It might probably pressure the metadata database with requests, each time the Scheduler parses the DAG recordsdata (defaults to thirty seconds).

It’s pretty simple to fall into this lure, until you perceive how the Scheduler parses DAGs and the way Variables are retrieved from the database.

Earlier than entering into the dialogue of how Variables are fetched from the metastore and what finest practices to use to be able to optimise DAGs , it’s necessary to get the fundamentals proper. For now, let’s simply deal with how we will really declare variables in Airflow.

As talked about already, there are a number of other ways to declare variables in Airflow. A few of them become safer and moveable than others, so let’s look at all and attempt to perceive their professionals and cons.

1. Making a variable from the Consumer Interface

On this first strategy, we’re going to create a variable by means of the Consumer Interface. From the highest menu choose Admin → Variables → +