Spark Performance Tunning in Short

Hello Friends, 

Hope you are doing good! 

As a Software Engineer if we are working into Data Engineering side then we should be focus more on job optimization so job performance can be enhanced to process a good amount of data very smoothly within the defined timeline. Spark optimizations are crucial for improving the performance of Spark jobs.  In this post let's focus on few points of Apache Spark Optimizations which can help to optimize our job to perform more better. Here's a concise overview of some key optimizations:

  1. 1. Partitioning: Ensure that data is evenly distributed across partitions. Use repartition() and coalesce() to adjust the number of partitions as needed.


  3. 2. Caching and Persistence: Cache frequently accessed RDDs or DataFrames using cache() or persist() to avoid recomputation.


  5. 3. Shuffling: Minimize data shuffling between nodes by using operations like reduceByKey, aggregateByKey, and combineByKey rather than groupByKey.


  7. 4. Broadcast Variables: Use broadcast variables for read-only variables that are large and need to be shared across tasks.


  9. 5. Predicate Pushdown: Push filters as early as possible in your transformations to reduce the amount of data processed.


  11. 6. Column Pruning: Only select columns that are necessary for your operations to reduce I/O and memory usage.


  13. 7. Tuning Parallelism: Adjust the level of parallelism (e.g., spark.default.parallelism, spark.sql.shuffle.partitions) based on the size of your cluster and workload.


  15. 8. Data Serialization: Use efficient serialization formats like Kryo (spark.serializer) instead of Java serialization to reduce overhead.


  17. 9. Memory Management: Tune memory configurations such as executor memory, driver memory, and memory fractions (e.g., spark.executor.memory, spark.driver.memory, spark.memory.fraction).


  19. 10. Optimized Joins: When performing joins, ensure that the smaller dataset is broadcasted if one of the datasets is significantly smaller to reduce shuffle data.


  21. 11. Avoiding UDFs: Minimize the use of User-Defined Functions (UDFs) in Spark SQL as they can be less optimized compared to built-in functions.


  23. 12. Partitioning Strategy: Choose appropriate partitioning strategies based on the nature of your data and operations (e.g., hash partitioning, range partitioning).

By incorporating these optimizations, we can significantly enhance the performance and efficiency of our Spark applications.


Hope you enjoyed while reading these optimization tips. If you like then please Like, Comment and Share.


Thank You!


Comments

Post a Comment

Popular posts from this blog

Transformations and Actions in Spark

Image
Hello Friends, Today I’ll try to explain about Transformations and Actions in Spark. We know that Basically, Spark  RDD supports two types of Operations :          ·    Transformations          ·    Actions Transformations : Transformations are kind of operations which will transform our RDD data from one form to another form, and when we apply this operation on any RDD, we get a new RDD with transformed data   as RDDs in Spark are immutable. Operations like map, flatMap, filter, join, union are transformations. Now there is a point to be noted here and that is when we apply the transformation on any RDD it will not perform the operation immediately. It creates a DAG(Directed Acyclic Graph) using the applied operations, source RDD and functions used for transformation. And it keeps on building this graph using the references till we apply any action operation on the last lined up R...
Read more

Knowledge about Apache Sqoop and its all basic commands to import and export the Data

Sqoop is a command-line interface application for transferring data between Relational Databases and Hadoop. We can say that Sqoop is a connector between RDBMS to Hadoop (Import) or Hadoop to RDBMS(Export). Options while importing data to Hadoop: .        Importing table data from RDBMS table to HDFS(file system) .        Importing table data from RDBMS table to Hive table .        Importing table data from RDBMS table to HBase  Options while exporting data to RDBMS:  .        Exporting data from HDFS(file system) to RDBMS table  .        Export data from Hadoop (Hive) to RDBMS table List all databases in MySQL sqoop-list-databases --connect jdbc:mysql://localhost --username hadoop --password hadoop List all tables in MySQL sqoop-list-tables --connect jdbc:mysql://localhost/hive --username hadoop --P ...
Read more

How to Convert a Spark DataFrame String Type Column to Array Type and Split all the json files of this column into rows : Part - 1

Image
Hi Friends, In this post, I'd like to explore a project scenario of json data. Suppose, We are getting a DataFrame from Source which has a column ArrayOfJsonStrings, which is actually an Array of Json files/data, but Data Type of this Column is String. We need to Split All the json files of this ArrayOfJsonStrings column into possible number of rows. Below is the Input and Output DataFrames : Input DataFrame : Output DataFrame : Below is the code for the same with explained steps : import org.apache.spark.SparkConf import org.apache.spark.sql.SparkSession import org.apache.spark.sql.functions._ object SplitArrayOfJsonsToRows {   def main(args: Array[String]) {     //Creating SparkSession     lazy val conf = new SparkConf().setAppName("split-array-of-json-to-row").set("spark.default.parallelism", "1")       .setIfMissing("spark.master", "local[*]")     lazy val sparkSession = SparkSession.builder()...
Read more