How to Convert Json data/String to DataFrame in Spark.


Hi Friends,

Today, I'd like to show that how we can convert a json data to DataFrame for further use.
This is a very simple and small use case, but it can be helpful for learners.


Input Json String :  {"file_name":"test_20200202122754.json","object_class":"Monitor","object_class_instance":"Monitor","relation_tree":"Source~>HD_Info~>Monitor","Monitor":{"Index":"0","Vendor_Data":"58F5Y","Monitor_Type":"Lenovo Monitor","HInfoID":"650FEC74"}}

Expected Output DataFrame :

Below is code for the same :

 import org.apache.spark.{SparkConf, SparkContext}
import org.apache.spark.sql.SparkSession


 object ConvertJsonToDataFrame extends App {

   //Creating SparkSession
  lazy val conf = new SparkConf().setAppName("json-to-DataFrame").set("spark.default.parallelism", "2")
    .setIfMissing("spark.master", "local[*]")
  lazy val sparkSession = SparkSession.builder().config(conf).getOrCreate()
  lazy val sparkContext: SparkContext = sparkSession.sparkContext
  import sparkSession.implicits._

   //Raw Json Data to test
  val jsonStr = """{"file_name":"test_20200202122754.json","object_class":"Monitor","object_class_instance":"Monitor","relation_tree":"Source~>HD_Info~>Monitor","Monitor":{"Index":"0","Vendor_Data":"58F5Y","Monitor_Type":"Lenovo Monitor","HInfoID":"650FEC74"}}"""

   //Loading the Json Data to create a DataFrame
  val jsonDF = sparkSession.read.json(Seq(jsonStr).toDS)

   jsonDF.show(false)

   //Fetch all the Column;s Value and Drop the Nested Column after extract it.
  val getAllColumns = jsonDF.select($"*", $"Monitor.*").drop("Monitor")
  getAllColumns.show(false)

 }




Highlighted Output as expected :


I Hope, This Post was helpful, please do like, comment and share.
Thank You !

Comments

Post a Comment

Popular posts from this blog

Knowledge about Apache Sqoop and its all basic commands to import and export the Data

Sqoop is a command-line interface application for transferring data between Relational Databases and Hadoop. We can say that Sqoop is a connector between RDBMS to Hadoop (Import) or Hadoop to RDBMS(Export). Options while importing data to Hadoop: .        Importing table data from RDBMS table to HDFS(file system) .        Importing table data from RDBMS table to Hive table .        Importing table data from RDBMS table to HBase  Options while exporting data to RDBMS:  .        Exporting data from HDFS(file system) to RDBMS table  .        Export data from Hadoop (Hive) to RDBMS table List all databases in MySQL sqoop-list-databases --connect jdbc:mysql://localhost --username hadoop --password hadoop List all tables in MySQL sqoop-list-tables --connect jdbc:mysql://localhost/hive --username hadoop --P ...
Read more

Transformations and Actions in Spark

Image
Hello Friends, Today I’ll try to explain about Transformations and Actions in Spark. We know that Basically, Spark  RDD supports two types of Operations :          ·    Transformations          ·    Actions Transformations : Transformations are kind of operations which will transform our RDD data from one form to another form, and when we apply this operation on any RDD, we get a new RDD with transformed data   as RDDs in Spark are immutable. Operations like map, flatMap, filter, join, union are transformations. Now there is a point to be noted here and that is when we apply the transformation on any RDD it will not perform the operation immediately. It creates a DAG(Directed Acyclic Graph) using the applied operations, source RDD and functions used for transformation. And it keeps on building this graph using the references till we apply any action operation on the last lined up R...
Read more

Data Lake, Data Warehouse, Data Mart, and Delta Lake

🏦 Data Lake, Data Warehouse, Data Mart, and Delta Lake Explained through a story you won't forget — using banking data 💳 👨‍💼 Meet Arjun. He works at a bank and wants to understand customer behavior to improve services. So, he starts collecting everything the bank has: ATM withdrawal logs Mobile app clickstream events Loan application forms Voice call recordings with customer care Account transaction history PDFs, emails, scanned documents He stores it all in one central location without worrying about format or structure. 👉 This is a Data Lake – a huge storage system where raw data of all types (structured, semi-structured, unstructured) is dumped for future use. Now, Arjun’s analytics team needs clean, trusted data to build dashboards and generate reports. So, he processes the raw data: He filters out errors, standardizes formats, fills missing values, and organizes it into structured formats (like SQL tables or clean Parquet files) — optimize...
Read more