How to Convert Map Data to Spark DataFrame in Scala

Hi Friends,

 In this post, I'd like to explore that how to create DataFrame from Map Data as below two use cases  :

Input Map Data :

 Map(Id -> 111, Name -> Anamika, City -> Bangalore, State -> Karnataka)
Map(Id -> 222, Name -> Divyanshu, City -> Bangalore, State -> Karnataka)
Map(Id -> 333, Name -> Himanshu, City -> Gandhi Nagar, State -> Gujrat)
Map(Id -> 444, Name -> Priyanshu, City -> Allahabad, State -> Uttar Pradesh)
Map(Id -> 555, Name -> Pranjal, City -> Allahabad, State -> Uttar Pradesh)


Case 1. Create DataFrame from Map Data and mapping with keys as column names

Expected Output for case 1 : 


Case 2. Create DataFrame (Map Columns [key, value]) from Map Data and mapping with keys as column names  

Expected Output for case 2 :
Below is the code with explanation :

 import org.apache.spark.sql.SparkSession
import org.apache.spark.{SparkConf, SparkContext}
import org.apache.spark.rdd.RDD

 object ConvertMapToDataFrame extends App {
  //Creating SparkSession
  lazy val conf = new SparkConf().setAppName("map-to-dataframe").setIfMissing("spark.master", "local[*]")
  lazy val sparkSession = SparkSession.builder().config(conf).getOrCreate()
  lazy val sparkContext: SparkContext = sparkSession.sparkContext
  import sparkSession.implicits._

   //Creating Input Map to Convert into DataFrame
  val rawMapRDD: RDD[Map[String, String]] = sparkContext.parallelize(Seq(
    Map("Id" -> "111", "Name" -> "Anamika", "City" -> "Bangalore", "State" -> "Karnataka"),
    Map("Id" -> "222", "Name" -> "Divyanshu", "City" -> "Bangalore", "State" -> "Karnataka"),
    Map("Id" -> "333", "Name" -> "Himanshu", "City" -> "Gandhi Nagar", "State" -> "Gujrat"),
    Map("Id" -> "444", "Name" -> "Priyanshu", "City" -> "Allahabad", "State" -> "Uttar Pradesh"),
    Map("Id" -> "555", "Name" -> "Pranjal", "City" -> "Allahabad", "State" -> "Uttar Pradesh")))

   //Printing above RDD : rawMapRDD
  rawMapRDD.foreach(println)
  //Getting column names from RDD to create DataFrame
  val getColumns = rawMapRDD.take(1).flatMap(x => x.keys)

   //Printing column names
  getColumns.foreach(println)

   //Case 1. Creating DataFrame from Map Data and mapping with column names from getColumns
  val finalDF = rawMapRDD.map { value =>
    val dataList = value.values.toList
    (dataList(0), dataList(1), dataList(2), dataList(3))
  }.toDF(getColumns: _*)

   //Printing DataFrame
  finalDF.show(false)
  
  //Case 2. Creating DataFrame (Map Columns [key, value]) from Map Data and mapping with column names from getColumns
  val finalMapDF = rawMapRDD.map { value =>
    val dataList = value.toList
    (dataList(0), dataList(1), dataList(2), dataList(3))
  }.toDF(getColumns: _*)

  //Printing Map Column's DataFrame
  finalMapDF.show(false)
}
Output DataFrames from above code :





I hope, this post was helpful. Please do like, comment and share.

 Thank You!

Comments

Post a Comment

Popular posts from this blog

Transformations and Actions in Spark

Image
Hello Friends, Today I’ll try to explain about Transformations and Actions in Spark. We know that Basically, Spark  RDD supports two types of Operations :          ·    Transformations          ·    Actions Transformations : Transformations are kind of operations which will transform our RDD data from one form to another form, and when we apply this operation on any RDD, we get a new RDD with transformed data   as RDDs in Spark are immutable. Operations like map, flatMap, filter, join, union are transformations. Now there is a point to be noted here and that is when we apply the transformation on any RDD it will not perform the operation immediately. It creates a DAG(Directed Acyclic Graph) using the applied operations, source RDD and functions used for transformation. And it keeps on building this graph using the references till we apply any action operation on the last lined up RDD. That is why the transformation in Spark are lazy. After the transformatio
Read more

How to convert XML String to JSON String in Spark-Scala

Image
Hello Friends, Today I'd like to share the method for conversion of XML data to Json data using Scala. This is done with org.json library which we use  to encode or decode  JSON  data. Input XML : <Summary_Log> <Monitor>        <HardwareInfo>FD2F</HardwareInfo>       <Id>0</Id>       <Monitor_Type>LCD</Monitor_Type>       <Vendor_Data>110HL</Vendor_Data> </Monitor> <Processor>       <HardwareInfo>FD2F</HardwareInfo>      <Id>1</Id>        <Used>80</Used>      <Utilization>19</Utilization> </Processor> </Summary_Log> Output Json : {"Summary_Log": {     "Monitor": {         "Monitor_Type": "LCD",         "Vendor_Data": "110HL",         "HardwareInfo": "FD2F",         "Id": 0     },     "Processor": {        
Read more

How to Convert a Spark DataFrame to Map in Scala

Image
Hi Friends, In this post, I am going to explain that how we can convert Spark DataFrame to Map. Input DataFrame : Output map : Below is the explained code with all the steps along with its Output.   import org.apache.spark.SparkConf import org.apache.spark.sql.{Row, SparkSession} import org.apache.spark.sql.types.{IntegerType, StringType, StructField, StructType} object ConvertDFToMap {   def main(args: Array[String]) {     //Creating SparkSession     lazy val conf = new SparkConf().setAppName("DataFrame-to-Map").set("spark.default.parallelism", "1")       .setIfMissing("spark.master", "local[*]")     lazy val sparkSession = SparkSession.builder().config(conf).getOrCreate()     // Creating Seq of Raw Data to create DataFrame to test     val rawData = Seq(       Row(1, "Anamika", "Uttar Pradesh"),       Row(2, "Ananya", "Delhi"),       Row(3, "Ambika",
Read more