I’ve been working with Airflow for greater than three years now and total, I’m fairly assured with it. It’s a strong orchestrator that helps me construct information pipelines rapidly and in a scalable vogue whereas for many issues I’m seeking to implement it comes with batteries included.
Just lately, and whereas getting ready myself to get a certification for Airflow, I’ve come throughout many alternative issues I had actually no clue about. And this was primarily my motivation to jot down this text and share with you a couple of Airflow internals which have completely blown my thoughts!
1. Scheduler solely parses information containing sure key phrases
The Airflow Scheduler will parse solely information containing airflow or dag within the code! Sure, you’ve heard this proper! If a file beneath the DAG folder doesn’t include no less than certainly one of these two key phrases, it should merely not be parsed by the scheduler.
If you wish to modify this rule such that that is not a requirement for the scheduler, you may merely set DAG_DISCOVERY_SAFE_MODE configuration setting to False. In that case, the scheduler will parse all information beneath your DAG folder (/dags).
I wouldn’t suggest disabling this test although, since doing so doesn’t actually make any sense. A correct DAG file may have Airflow imports and DAG definition which suggests the necessities for parsing that file are met) however it’s value figuring out that this rule exists.
2. Variables with sure key phrases of their identify have their values hidden
We all know that by default, Airflow will conceal delicate data saved in a Connection (and extra particularly within the password area), however what about Variables?
Properly, that is certainly potential and the thoughts blowing factor is that Airflow can do that robotically for you. If a variable accommodates sure key phrases, that may presumably point out delicate data, then its worth will robotically be hidden.
Right here’s an inventory of key phrases that may make a Variable qualify for having delicate data retailer as…