Big Data End to End Pipeline Design on major Cloud Platforms.

Simple Process for designing a Pipeline can be like below. Let's understand it with examples:

Ingest -> Store -> Process -> Serve.

Ingest => Get the data from multiple sources using some ingestion framework.

Example => AWS Glue, Azure Data Factory, Sqoop and NiFi

Store => Since we are going to store huge amount of data we need a Distributed/Object store.

Example => HDFS, Amazon S3, Azure blob storage and Google Cloud Storage (GCS)

Process => Since our data is sitting on multiple machines our traditional programming styles won't work.                        we need a distributed processing framework.

Example => Apache Spark, MapReduce

*Apache Spark can be on-premises, Databricks etc.

Serve => The processed Data now needs to show on UI, Visualization for reporting.

Example => NoSQL DB's like HBase, MongoDB, DynamoDB, CosmosDB, Cassandra or even RDBMS databases.

Let's see this pipeline across 2 major public cloud providers AWS and Azure including On-Premise :

AWS :
Glue -> Amazon S3 -> Athena/Redshift -> DynamoDB

Azure : 
Data Factory -> ADLS Gen2 -> Databricks -> Delta/Synapse

On Premise :
Sqoop/NiFi -> HDFS -> Apache Spark -> HBase/Hive

Note => There can be many other ways to Design these pipelines. Some of them are as below :
1.  Glue --> S3 --> Apache Spark/Athena --> S3/DynamoDB
2.  Data source --> ingest --> S3 --> Snowflake (through Snowpipe) --> Transformations in one layer of DB --> Power BI

Hope you enjoyed while reading these optimization tips. If you like then please Like, Comment and Share.


Thank You!

Comments

Post a Comment

Popular posts from this blog

Transformations and Actions in Spark

Image
Hello Friends, Today I’ll try to explain about Transformations and Actions in Spark. We know that Basically, Spark  RDD supports two types of Operations :          ·    Transformations          ·    Actions Transformations : Transformations are kind of operations which will transform our RDD data from one form to another form, and when we apply this operation on any RDD, we get a new RDD with transformed data   as RDDs in Spark are immutable. Operations like map, flatMap, filter, join, union are transformations. Now there is a point to be noted here and that is when we apply the transformation on any RDD it will not perform the operation immediately. It creates a DAG(Directed Acyclic Graph) using the applied operations, source RDD and functions used for transformation. And it keeps on building this graph using the references till we apply any action operation on the last lined up R...
Read more

Knowledge about Apache Sqoop and its all basic commands to import and export the Data

Sqoop is a command-line interface application for transferring data between Relational Databases and Hadoop. We can say that Sqoop is a connector between RDBMS to Hadoop (Import) or Hadoop to RDBMS(Export). Options while importing data to Hadoop: .        Importing table data from RDBMS table to HDFS(file system) .        Importing table data from RDBMS table to Hive table .        Importing table data from RDBMS table to HBase  Options while exporting data to RDBMS:  .        Exporting data from HDFS(file system) to RDBMS table  .        Export data from Hadoop (Hive) to RDBMS table List all databases in MySQL sqoop-list-databases --connect jdbc:mysql://localhost --username hadoop --password hadoop List all tables in MySQL sqoop-list-tables --connect jdbc:mysql://localhost/hive --username hadoop --P ...
Read more

How to Convert a Spark DataFrame String Type Column to Array Type and Split all the json files of this column into rows : Part - 1

Image
Hi Friends, In this post, I'd like to explore a project scenario of json data. Suppose, We are getting a DataFrame from Source which has a column ArrayOfJsonStrings, which is actually an Array of Json files/data, but Data Type of this Column is String. We need to Split All the json files of this ArrayOfJsonStrings column into possible number of rows. Below is the Input and Output DataFrames : Input DataFrame : Output DataFrame : Below is the code for the same with explained steps : import org.apache.spark.SparkConf import org.apache.spark.sql.SparkSession import org.apache.spark.sql.functions._ object SplitArrayOfJsonsToRows {   def main(args: Array[String]) {     //Creating SparkSession     lazy val conf = new SparkConf().setAppName("split-array-of-json-to-row").set("spark.default.parallelism", "1")       .setIfMissing("spark.master", "local[*]")     lazy val sparkSession = SparkSession.builder()...
Read more