Airflow dag variables code#

XComs can be used to pass small pieces of information between Airflow tasks.Params are not encrypted and should not be used to store secrets. You can define defaults for params at the DAG or task level and override them at runtime. Params can be used to store information specific to a DAG or DAG run.They are also often used to store credentials for local development. One advantage of using environment variables is that you can include their creation in your CI/CD process. Environment variables are very versatile, as they can be used to both store arbitrary information and configure Airflow. There is no direct way to see environment variables in the Airflow UI but they can be accessed using os.getenv('MY_ENV_VAR') inside of Airflow DAGs and tasks. Environment variables store small pieces of information that are available to the whole Airflow environment.The ideal option depends on what type of information you are storing and where and how you want to access it:

Airflow dag variables code#

See the Airflow documentation for examples of code showing good and bad practices for accessing Airflow variables in a DAG.Īside from Airflow variables, there are other ways of storing information in Airflow. To mask Airflow variables in the UI and logs, include a substring indicating a sensitive value in your Airflow variable name. Airflow variables are encrypted with Fernet when they are written to the Airflow metastore.If you do use Airflow variables in top-level DAG code, use the Jinja template syntax so that your Airflow variables are only rendered when a task executes.See DAG writing best practices in Apache Airflow. You should avoid using Airflow variables outside of tasks in top-level DAG code, as they will create a connection to the Airflow metastore every time the DAG is parsed, which can lead to performance issues.Airflow variables should be used for information that is runtime dependent but doesn't change too frequently.There are some best practices to keep in mind when using Airflow variables: They are Airflow’s runtime configuration concept and defined using the object. See Introduction to Airflow DAGs.īest practices for storing information in Airflow Īirflow variables store key-value pairs or short JSON objects that need to be accessible in your whole Airflow instance. To get the most out of this guide, you should have an understanding of: Astronomer Academy: Airflow: Variables 101 module.There are multiple resources for learning about this topic.