In this tutorial I will go over the details on how to install and run Apache Airflow as a Systemd service on an AWS EC2 instance and orchestrate a cluster. I will also describe how to use a MySQL database as a backend for Airflow. Finally, I will show how to execute ssh commands across the cluster by going over a simple example of the Airflow SSH operator.
Ride-Share-Compare was the project that I completed as part of the Insight Data Engineering program. It is a platform where users can compare different ride-share options without the need to check each ride-share provider's application.