Types of File Formats in Hadoop with example.

Hadoop supports various file formats to store data efficiently, ensuring optimized storage and processing. Here's an expanded explanation of each type with examples:

  1. 1. TextFile:

    • Description: This is a basic, plain-text storage format where each line in the file represents a record.
    • Example:
      • Anamika Singh,Bangalore,35 Shekhar,Delhi,30
    • 5

  2. 2. SequenceFile:

    • Description: SequenceFile is a binary file format that stores key-value pairs. It's designed to be splittable and supports various compression codecs.
    • Example:
    • key1 value1 key2 value2
    • Key1 Value1 Key2 Value2

  3. 3. Avro:

  4. 4. Parquet:

    • Description: Parquet is a columnar storage file format, optimized for performance and space efficiency. It supports complex nested data structures and provides efficient compression.
    • Example: The data in Parquet is binary, so it won't be readable like text formats.


  6. 5. ORC (Optimized Row Columnar):

    • Description: ORC is another columnar storage file format similar to Parquet. It's designed for high performance and offers efficient compression, making it suitable for large-scale data processing.
    • Example: The data in ORC is stored in a binary format.


  8. 6. JSON:

    • Description: JSON (JavaScript Object Notation) is a human-readable text-based format for data interchange. It's widely used for data transmission between systems.
    • Example:
      {"name": "Anamika Singh", "age": 35, "city": "New Delhi"}
{"name": "Ram", "age": 30, "city": "Bangalore"}


  10. 7. XML:

    • Description: XML (eXtensible Markup Language) is a markup language that encodes data in a text format. It's used for storing and transporting structured data.
    • Example:
    • <person> <name>Anamika Singh</name> <age>35</age> <city>New Delhi</city> </person>

  11. 8. Hive RCFile (Record Columnar File):

    • Description: RCFile is a columnar storage file format developed for Apache Hive. It stores rows in a columnar format to improve query performance.
    • Example: The data is stored in a binary format optimized for Hive queries.

Each of these file formats has its advantages and is suitable for specific use cases. When choosing a file format in Hadoop, consider factors such as data size, query performance, schema flexibility, and compression requirements.


Hope you enjoyed while reading these optimization tips. If you like then please Like, Comment and Share.


Thank You!


Comments

Post a Comment

Popular posts from this blog

Transformations and Actions in Spark

Image
Hello Friends, Today I’ll try to explain about Transformations and Actions in Spark. We know that Basically, Spark  RDD supports two types of Operations :          ·    Transformations          ·    Actions Transformations : Transformations are kind of operations which will transform our RDD data from one form to another form, and when we apply this operation on any RDD, we get a new RDD with transformed data   as RDDs in Spark are immutable. Operations like map, flatMap, filter, join, union are transformations. Now there is a point to be noted here and that is when we apply the transformation on any RDD it will not perform the operation immediately. It creates a DAG(Directed Acyclic Graph) using the applied operations, source RDD and functions used for transformation. And it keeps on building this graph using the references till we apply any action operation on the last lined up R...
Read more

Knowledge about Apache Sqoop and its all basic commands to import and export the Data

Sqoop is a command-line interface application for transferring data between Relational Databases and Hadoop. We can say that Sqoop is a connector between RDBMS to Hadoop (Import) or Hadoop to RDBMS(Export). Options while importing data to Hadoop: .        Importing table data from RDBMS table to HDFS(file system) .        Importing table data from RDBMS table to Hive table .        Importing table data from RDBMS table to HBase  Options while exporting data to RDBMS:  .        Exporting data from HDFS(file system) to RDBMS table  .        Export data from Hadoop (Hive) to RDBMS table List all databases in MySQL sqoop-list-databases --connect jdbc:mysql://localhost --username hadoop --password hadoop List all tables in MySQL sqoop-list-tables --connect jdbc:mysql://localhost/hive --username hadoop --P ...
Read more

How to Convert a Spark DataFrame String Type Column to Array Type and Split all the json files of this column into rows : Part - 1

Image
Hi Friends, In this post, I'd like to explore a project scenario of json data. Suppose, We are getting a DataFrame from Source which has a column ArrayOfJsonStrings, which is actually an Array of Json files/data, but Data Type of this Column is String. We need to Split All the json files of this ArrayOfJsonStrings column into possible number of rows. Below is the Input and Output DataFrames : Input DataFrame : Output DataFrame : Below is the code for the same with explained steps : import org.apache.spark.SparkConf import org.apache.spark.sql.SparkSession import org.apache.spark.sql.functions._ object SplitArrayOfJsonsToRows {   def main(args: Array[String]) {     //Creating SparkSession     lazy val conf = new SparkConf().setAppName("split-array-of-json-to-row").set("spark.default.parallelism", "1")       .setIfMissing("spark.master", "local[*]")     lazy val sparkSession = SparkSession.builder()...
Read more