How To Convert a DataFrame to Map Column in Spark-Scala.

Hi Friends,

In this post, I'd like to explore a small use case of convert/create Spark DataFrame to Map.
I'll use Spark map function to get it. Apparently the precondition is, we must have all the columns with same type otherwise we will get spark error. in This example all columns are String Type.

 Input DataFrame :


Output DataFrame :

 1.  Create a Map column in existed DataFrame, Map key should be column Name, Map Value should be column value.

Example output with created new column map_value :

2.  Create a Map column with Nested Map in existed DataFrame, for All Map values, key should be column Name and for one Nested Map Value, key should be another column value instead of column name.

Example output with created new column map_value :
Below is the code with output  for above each use cases :

 import org.apache.spark.SparkConf
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.functions._

 object DfToMap extends App {

   //Creating SparkSession
  lazy val conf = new SparkConf().setAppName("df-to-map").set("spark.default.parallelism", "2")
    .setIfMissing("spark.master", "local[*]")
  lazy val sparkSession = SparkSession.builder().config(conf).getOrCreate()
  import sparkSession.implicits._

   //Creating raw DataFrame
  val rawDF = Seq(("Samsung", "L", "68", "Delhi", "India"), ("Sony", "XL", "100", "Tokyo", "Japan"))
    .toDF("brand", "size", "sales", "city", "country")

   rawDF.show(false)
  //Final Map with created Map Column (map_value) where Column_Name as Key and Value as Value
  val finalMapwithAllColumnsAsKey = rawDF.withColumn("map_value",map(lit("brand"),$"brand",lit("size"),$"size",lit("sales"),$"sales",
    lit("city"),$"city",lit("country"), $"country"))

   finalMapwithAllColumnsAsKey.show(false)

   //Final Map with created Map Column (map_value) where Column_Name as Key and Value as Value along with a Nested Map where Column Value as Key of this Map.
  val finalMapwithCountryValueAsKeyForInnermap = rawDF.withColumn("map_value",map(lit("brand"),$"brand",lit("size"),$"size",lit("sales"),$"sales",
    lit("city"),$"city",lit("country"), $"country", $"country",map(lit("brand"),$"brand",lit("size"),$"size",lit("sales"),$"sales",
    lit("city"),$"city",lit("country"), $"country")
   .cast("string"))).select("map_value")

   finalMapwithCountryValueAsKeyForInnermap.show(false)
}


Output with above code :





I Hope, This post was helpful, please do like, comment and share.  

Thank You!

Comments

Post a Comment

Popular posts from this blog

Transformations and Actions in Spark

Image
Hello Friends, Today I’ll try to explain about Transformations and Actions in Spark. We know that Basically, Spark  RDD supports two types of Operations :          ·    Transformations          ·    Actions Transformations : Transformations are kind of operations which will transform our RDD data from one form to another form, and when we apply this operation on any RDD, we get a new RDD with transformed data   as RDDs in Spark are immutable. Operations like map, flatMap, filter, join, union are transformations. Now there is a point to be noted here and that is when we apply the transformation on any RDD it will not perform the operation immediately. It creates a DAG(Directed Acyclic Graph) using the applied operations, source RDD and functions used for transformation. And it keeps on building this graph using the references till we apply any action operation on the last lined up RDD. That is why the transformation in Spark are lazy. After the transformatio
Read more

How to Convert a Spark DataFrame to Map in Scala

Image
Hi Friends, In this post, I am going to explain that how we can convert Spark DataFrame to Map. Input DataFrame : Output map : Below is the explained code with all the steps along with its Output.   import org.apache.spark.SparkConf import org.apache.spark.sql.{Row, SparkSession} import org.apache.spark.sql.types.{IntegerType, StringType, StructField, StructType} object ConvertDFToMap {   def main(args: Array[String]) {     //Creating SparkSession     lazy val conf = new SparkConf().setAppName("DataFrame-to-Map").set("spark.default.parallelism", "1")       .setIfMissing("spark.master", "local[*]")     lazy val sparkSession = SparkSession.builder().config(conf).getOrCreate()     // Creating Seq of Raw Data to create DataFrame to test     val rawData = Seq(       Row(1, "Anamika", "Uttar Pradesh"),       Row(2, "Ananya", "Delhi"),       Row(3, "Ambika",
Read more

How to Handle and Convert DateTime format in Spark-Scala.

Image
Hi Friends, In this post, I'd like to explain that how to handle and convert the date format in expected date format. Below I'm providing few examples : 1. Change Date format to yyyy-mm-dd from dd-mm-yyyy or mm-dd-yyyy : Input Date :  03-04-2020T15:26:51+05:30 Converted Output : 2020-04-03T15:26:051 Code to get the same output : def changeDateFormat (inputDateString: String ) = { var convertedDateFormat = "" try { val inputDateFormat: SimpleDateFormat = new SimpleDateFormat( "dd-MM-yyyy'T'HH:mm:sss" ) val date = inputDateFormat.parse(inputDateString) val df = new SimpleDateFormat( "yyyy-MM-dd'T'HH:mm:sss" ) convertedDateFormat = df.format(date) } catch { case ex: Exception => { "Exception is : " + ex } logger .info( "[HandleDateTimeUtils] Exception while converting time to 'yyyy-MM-dd'T'HH:mm:sss' Format. Input is: " + inputDateString , ex.getMessag
Read more