Information about scheduling tools & Databricks

 Topics Covered:

●What is Apache Airflow?

●Features of Apache Airflow

●Components of Apache Airflow

●What is Databricks?

●What is Data Lakes?

●Features of a Data Lake

●Differences between a Data Lake and Data Warehouse

●Applications of Databricks

What is Apache Airflow?

●Automation of work plays a key role in any industry and it is one of the quickest ways to reach functional efficiency.

●Most of us have to deal with different work flows like collecting data from multiple databases, preprocessing it, uploading it, and reporting it.

●Consequently, it would be great if our daily tasks just automatically trigger on defined time, and all the processes get executed in order. This is where Apache Airflow comes into picture.

●Apache Air flow is an open-source platform for developing, scheduling, and monitoring batch-oriented workflows

●Airflow’s extensible Python frame work enables you to build workflows connecting with virtually any technology.

●A web interface helps manage the state of your workflows. Air flow is deploy able in many ways, varying from a single process on your laptop to a distributed setup to support even the biggest workflows.

●Apache Airflow is a workflow engine that will easily schedule and run your complex data pipelines. It will make sure that each task of your data pipeline will get executed in the correct order and each task gets the required resources.

●For More information on Airflow, please refer to the Airflow Official Documentation

Features of Apache Airflow

●Easy to Use: If you have a bit of python knowledge, you are good to go and deploy on Airflow.

●Open Source: It is free and open-source with a lot of active users.

●Robust Integrations: It will give you ready to use operators so that you can work with Google Cloud Platform, Amazon AWS, Microsoft Azure, etc.

●Use Standard Python to code: You can use python to create simple to complex work flows with complete flexibility.

●Amazing User Interface: You can monitor and manage your workflows. It will allow you to check the status of completed and ongoing tasks.

Components of Apache Airflow

●DAG: It is the Directed Acyclic Graph–a collection of all the tasks that you want to run which is organized and shows the relationship between different tasks. It is defined in a python script.

●Web Server: It is the user interface built on the Flask. It allows us to monitor the status of the DAGs and trigger them. 

- Meta data Database: Airflow stores the status of all the tasks in a data base and does all read/write operations of a work flow from here.

●Scheduler: As the name suggests, this component is responsible for scheduling the execution of DAGs. It retrieves and updates the status of the task in the database

What is Databricks?

It was founded by the creators of Apache Spark. It provides automated cluster and workflow management.
● Databricks is basically a cloud-based data engineering tool that is widely used by companies to process and transform large quantities of data and explore the data.
● This is used to process and transform extensive amounts of data and explore it through machine learning models. It allows organizations to quickly achieve the full potential of combining their data, ETL processes, and machine learning.
● Databricks in simple terms is a data warehousing, machine learning web-based platform developed by the creators of Spark.
● It’s a one-stop product for all data needs, from data storage, analysis data and derives insights using Spark SQL, build predictive models using Spark ML, it also provides active connections to visualization tools such as Power BI, Tableau, QlikView, etc.
● Databricks is integrated with Amazon Web Services, Microsoft Azure, and Google Cloud Platform making it easy to integrate with these major cloud computing infrastructures.

For more information on Databricks, please refer to the Databricks Official Documentation. 

Example: Consider a loan approval batch pipeline that triggers every night at 9 PM with 100M applications. Conventional models take days to evaluate the input data and provide an output, which leads to operational delays and unhappy customers, case in point banking systems. What if the process is completed on the go, and the loan is approved in seconds as in fintech apps? This is what Databricks brings to the table with an array of products and solutions. Faster ETL and easier decision making.

What is Data Lakes?
● Data lake in Databricks is used for the storage layer which can accommodate structured or unstructured, streaming or batch information. It’s a simple platform to store all data.
● Data lake from Databricks is called Delta Lake.
● Data Lakes and Data Warehouses are established terms when it comes to storing Big Data, but the two terms are not synonymous. A data lake is a large pool of raw data for which no use has yet been determined. A data warehouse, on the other hand, is a repository for structured, filtered data that has already been processed for a specific purpose.

Features of a Data Lake
In a data lake, the data is ingested into a storage layer with minimal transformation while maintaining the input format, structure, and granularity. This contains both structured and unstructured data. This results in several features, such as:

Collection of multiple data sources, such as bulk data, external data, real-time data, and many more.
● Control of ingested data and focus on documenting data structure.
● Generally useful for analytical reports and data science.
● But can also include an integrated Data Warehouse to provide classic management reports and dashboards.
● A Data Lake is a data storage pattern that prioritizes availability over everything else, across the enterprise, across all departments, and for all users of the data.
● Easy integrability of the new data source.

Differences between a Data Lake and Data Warehouse
While data warehouses use the classic ETL process in combination with structured data in a relational database, a data lake uses paradigms such as ELT and a schema on read as well as unstructured data.

Features of a Delta Lake in Databricks
● Features of Delta Lakes
● It’s based on the Parquet file format.
● Compatible with Apache Spark.
● Versioning of data.
● ACID transactions – (Atomicity, Consistency, Isolation, Durability) ensure data durability and consistency.
● Supports deleting and inserting into tables using API’s.
● Able to query millions of files using Spark

Applications of Databricks
The Databricks workspace provides user interfaces for many core data tasks, including tools for the following:
● Interactive notebooks
● Workflows scheduler and manager
● SQL editor and dashboards
● Data ingestion and governance
● Data discovery, annotation, and exploration
● Compute management
● Machine learning (ML) experiment tracking
● ML model serving
● A feature store
● Source control with Git