How to optimize slow running Apache Spark jobs?

Dear Friends,

We all know that optimizing the slow running Apache Spark jobs can be a big challenge, but we can follow the below steps which can be very helpful to make our jobs run faster :

1. Check Spark UI for slow tasks, enable AQE for partition skew.

2. Optimize join strategies, consider broadcast joins for small datasets.

3. Ensure sufficient resources are allocated.

4. Verify number of DataFrame partitions for proper parallelism.

5. Mitigate GC delays with off-heap memory.

6. Monitor disk spills, allocate more memory per CPU core if needed.

7. Opt for hash aggregation over sort aggregation when aggregating.

8. Implement caching. (It saves intermediate results in an accessible place that is ready for fast recalls, instead of performing the same calculations over and over again. Spark provides an optimization mechanism to store the intermediate computation of a Spark DataFrame so they can be reused in subsequent actions using cache() and persist() methods. When you persist a dataset, each node stores its partitioned data in memory and reuses them in other actions on that dataset.)

9. Choose the right file format and compression technique.

Try to consider the above options and for sure these will help to optimize the slow running Spark jobs.



Hope you enjoyed while reading these optimization tips. If you like then please Like, Comment and Share.  


Thank You!



Comments

Post a Comment

Popular posts from this blog

Transformations and Actions in Spark

Image
Hello Friends, Today I’ll try to explain about Transformations and Actions in Spark. We know that Basically, Spark  RDD supports two types of Operations :          ·    Transformations          ·    Actions Transformations : Transformations are kind of operations which will transform our RDD data from one form to another form, and when we apply this operation on any RDD, we get a new RDD with transformed data   as RDDs in Spark are immutable. Operations like map, flatMap, filter, join, union are transformations. Now there is a point to be noted here and that is when we apply the transformation on any RDD it will not perform the operation immediately. It creates a DAG(Directed Acyclic Graph) using the applied operations, source RDD and functions used for transformation. And it keeps on building this graph using the references till we apply any action operation on the last lined up R...
Read more

Knowledge about Apache Sqoop and its all basic commands to import and export the Data

Sqoop is a command-line interface application for transferring data between Relational Databases and Hadoop. We can say that Sqoop is a connector between RDBMS to Hadoop (Import) or Hadoop to RDBMS(Export). Options while importing data to Hadoop: .        Importing table data from RDBMS table to HDFS(file system) .        Importing table data from RDBMS table to Hive table .        Importing table data from RDBMS table to HBase  Options while exporting data to RDBMS:  .        Exporting data from HDFS(file system) to RDBMS table  .        Export data from Hadoop (Hive) to RDBMS table List all databases in MySQL sqoop-list-databases --connect jdbc:mysql://localhost --username hadoop --password hadoop List all tables in MySQL sqoop-list-tables --connect jdbc:mysql://localhost/hive --username hadoop --P ...
Read more

How to Convert a Spark DataFrame String Type Column to Array Type and Split all the json files of this column into rows : Part - 1

Image
Hi Friends, In this post, I'd like to explore a project scenario of json data. Suppose, We are getting a DataFrame from Source which has a column ArrayOfJsonStrings, which is actually an Array of Json files/data, but Data Type of this Column is String. We need to Split All the json files of this ArrayOfJsonStrings column into possible number of rows. Below is the Input and Output DataFrames : Input DataFrame : Output DataFrame : Below is the code for the same with explained steps : import org.apache.spark.SparkConf import org.apache.spark.sql.SparkSession import org.apache.spark.sql.functions._ object SplitArrayOfJsonsToRows {   def main(args: Array[String]) {     //Creating SparkSession     lazy val conf = new SparkConf().setAppName("split-array-of-json-to-row").set("spark.default.parallelism", "1")       .setIfMissing("spark.master", "local[*]")     lazy val sparkSession = SparkSession.builder()...
Read more