https://store-images.s-microsoft.com/image/apps.255.05c0293c-fd87-48fe-8b94-ee8262de21d4.94b8f460-6c0a-40f3-8cc3-74270e83045a.c0e654bf-7af7-49fe-8dc0-4a7de86484a9

Apache Airflow

Data Science Dojo

Apache Airflow

Data Science Dojo

Monitor and manage the data pipelines and complex workflows using Apache Airflow.

Apache Airflow: Open Source

Data Science Dojo delivers data science education, consulting, and technical services to harvest the power of data.

About the offer:

Apache Airflow is a powerful open-source platform for authoring, scheduling, and monitoring data and computing workflows. It is a highly scalable distributed system and can be connected to various sources making it flexible. These features allow it to be used efficiently in the orchestration of complex workflow and data pipelining problems. Users can represent their workflows in the form of graphs where the nodes of DAG(Directed Acyclic Graphs) represent the task. Airflow pipelines are defined in Python, allowing for dynamic pipeline generation. Many Organizations use Apache Airflow for their workflow monitoring.

Who benefits from this offer:

Following can benefit from Apache Airflow:

  • Teams of Developers
  • Data Scientists
  • Data Engineers
  • Machine Learning Engineers
  • DevOps
  • And anyone else interested in data science tools

What is included in this offer:
  • Python 3.8: Use Python programming language for dynamic pipeline generation
  • Rich User Interface: Helps in monitoring, managing, and visualizing the workflows
  • Executor: CeleryExecutor
  • Database: PostgreSQL
  • Message Broker: Redis
  • Robust Integrations
  • Code console: To view the DAGs code
  • Visualization: Tree, Graph, and Gantt chart visualization for DAGs
  • Security and Accessibility: Role-based actions
Technical Specifications:

Apache Airflow throughput improves with more CPU cores, more RAM, and faster disks

  • Minimum memory: 8GB RAM
  • Minimum vCPU: 2 vCPUs
  • Operating System: Ubuntu 20.04

Apache Airflow supports the following Databases and Executors:

Databases:
  • SQLite
  • MySQL
  • PostgreSQL
Executors:
  • Sequential Executor
  • Local Executor
  • Celery Executor
  • Kubernetes Executor

The default port Apache Airflow listens to is 8080. You can access the web interface at http://yourip:8080 using the credentials:

  • username:airflow
  • password:airflow

You can view the current configuration of Airflow from the tab 'Admin > Configuraions' in the top menubar. (Note: this can be disabled by your Airflow admin for security reasons)

https://store-images.s-microsoft.com/image/apps.12431.05c0293c-fd87-48fe-8b94-ee8262de21d4.94b8f460-6c0a-40f3-8cc3-74270e83045a.16c19af7-32cf-43e0-acc4-839307df55fe
/staticstorage/0e3134ef/assets/videoOverlay_7299e00c2e43a32cf9fa.png
https://store-images.s-microsoft.com/image/apps.12431.05c0293c-fd87-48fe-8b94-ee8262de21d4.94b8f460-6c0a-40f3-8cc3-74270e83045a.16c19af7-32cf-43e0-acc4-839307df55fe
/staticstorage/0e3134ef/assets/videoOverlay_7299e00c2e43a32cf9fa.png
https://store-images.s-microsoft.com/image/apps.53426.05c0293c-fd87-48fe-8b94-ee8262de21d4.94b8f460-6c0a-40f3-8cc3-74270e83045a.259e8c6f-3eae-4fe9-97b4-76ec7c43fadc
/staticstorage/0e3134ef/assets/videoOverlay_7299e00c2e43a32cf9fa.png
https://store-images.s-microsoft.com/image/apps.36367.05c0293c-fd87-48fe-8b94-ee8262de21d4.94b8f460-6c0a-40f3-8cc3-74270e83045a.ca2ae443-219a-4791-bc0a-a44d5f011a83
/staticstorage/0e3134ef/assets/videoOverlay_7299e00c2e43a32cf9fa.png
https://store-images.s-microsoft.com/image/apps.39922.05c0293c-fd87-48fe-8b94-ee8262de21d4.94b8f460-6c0a-40f3-8cc3-74270e83045a.512a2de7-d9b6-4535-9c42-8d80802ebd94
/staticstorage/0e3134ef/assets/videoOverlay_7299e00c2e43a32cf9fa.png
https://store-images.s-microsoft.com/image/apps.54825.05c0293c-fd87-48fe-8b94-ee8262de21d4.94b8f460-6c0a-40f3-8cc3-74270e83045a.41f0bd3e-84d7-4b8a-8e45-ade7aa8d76a4
/staticstorage/0e3134ef/assets/videoOverlay_7299e00c2e43a32cf9fa.png
https://store-images.s-microsoft.com/image/apps.38805.05c0293c-fd87-48fe-8b94-ee8262de21d4.94b8f460-6c0a-40f3-8cc3-74270e83045a.8ba01511-f218-493f-b5dc-de555d2b720c
https://store-images.s-microsoft.com/image/apps.60230.05c0293c-fd87-48fe-8b94-ee8262de21d4.94b8f460-6c0a-40f3-8cc3-74270e83045a.72eb9dc5-c8e8-44d1-b680-fce7a41e7231
https://store-images.s-microsoft.com/image/apps.44698.05c0293c-fd87-48fe-8b94-ee8262de21d4.94b8f460-6c0a-40f3-8cc3-74270e83045a.3b6bc1f5-2902-45de-99be-022d1d81f966
https://store-images.s-microsoft.com/image/apps.30252.05c0293c-fd87-48fe-8b94-ee8262de21d4.94b8f460-6c0a-40f3-8cc3-74270e83045a.2bade3a4-0567-4ce6-8999-f9d2ceb105e7
https://store-images.s-microsoft.com/image/apps.53906.05c0293c-fd87-48fe-8b94-ee8262de21d4.94b8f460-6c0a-40f3-8cc3-74270e83045a.54874f43-49f5-4cbb-823d-51704f7f855e