In other words, how exactly is the transform_script used?It is actually quite straightforward. When the file gets downloaded from S3,…
Continue Readingairflow
Yet Another Scalable Apache Airflow With Docker Example Setup
The name of virtualenv of choice, in this case airflow_jupyterwill be used later — because we’d rather not clutter our workstation, we…
Continue ReadingPython & Big Data: Airflow & Jupyter Notebook with Hadoop 3, Spark & Presto
If youre interested in producing charts of data in Jupyter Notebook then have a look at the Visualising Data with…
Continue ReadingCustomising Airflow: Beyond Boilerplate Settings
$ screen $ airflow webserver -p 8080 The following will launch Airflows scheduler..$ screen $ airflow scheduler If you open…
Continue ReadingAirflow: Lesser Known Tips, Tricks, and Best Practises
As an Airflow variable can contain JSON value, you can store all your DAG configuration inside a single variable as…
Continue Reading