What happens if we delete a file in HDFS with replication factor as 3?

Hello Friends, 

In this post let's try to understand that what happens if we delete a file in HDFS with replication factor as 3 with below use cases :

Use Case 1: When trash property is not enabled?

When we delete a file from HDFS, it removes the metadata pointing to the blocks that is stored in NameNode. As there will be no reference to the blocks in metadata so they are deleted completely.

Use Case 2: When trash property is enabled?

In this case the file is moved in Trash directory so we can recover that file for a specific duration that was configured using "fs.trash.interval" in core-site.xml.

Use Case 3: When trash property is enabled and using -skipTrash parameter when deleting the file?

In this case the files are also deleted from trash as well, so no snapshot is present for that file i.e. there will be no reference to the blocks of that file to NameNode metadata. So the file will be removed completely.


Hope you enjoyed while reading these optimization tips. If you like then please Like, Comment and Share.


Thank You!

Comments

Post a Comment

Popular posts from this blog

Transformations and Actions in Spark

Image
Hello Friends, Today I’ll try to explain about Transformations and Actions in Spark. We know that Basically, Spark  RDD supports two types of Operations :          ·    Transformations          ·    Actions Transformations : Transformations are kind of operations which will transform our RDD data from one form to another form, and when we apply this operation on any RDD, we get a new RDD with transformed data   as RDDs in Spark are immutable. Operations like map, flatMap, filter, join, union are transformations. Now there is a point to be noted here and that is when we apply the transformation on any RDD it will not perform the operation immediately. It creates a DAG(Directed Acyclic Graph) using the applied operations, source RDD and functions used for transformation. And it keeps on building this graph using the references till we apply any action operation on the last lined up R...
Read more

Knowledge about Apache Sqoop and its all basic commands to import and export the Data

Sqoop is a command-line interface application for transferring data between Relational Databases and Hadoop. We can say that Sqoop is a connector between RDBMS to Hadoop (Import) or Hadoop to RDBMS(Export). Options while importing data to Hadoop: .        Importing table data from RDBMS table to HDFS(file system) .        Importing table data from RDBMS table to Hive table .        Importing table data from RDBMS table to HBase  Options while exporting data to RDBMS:  .        Exporting data from HDFS(file system) to RDBMS table  .        Export data from Hadoop (Hive) to RDBMS table List all databases in MySQL sqoop-list-databases --connect jdbc:mysql://localhost --username hadoop --password hadoop List all tables in MySQL sqoop-list-tables --connect jdbc:mysql://localhost/hive --username hadoop --P ...
Read more

How to Convert a Spark DataFrame String Type Column to Array Type and Split all the json files of this column into rows : Part - 1

Image
Hi Friends, In this post, I'd like to explore a project scenario of json data. Suppose, We are getting a DataFrame from Source which has a column ArrayOfJsonStrings, which is actually an Array of Json files/data, but Data Type of this Column is String. We need to Split All the json files of this ArrayOfJsonStrings column into possible number of rows. Below is the Input and Output DataFrames : Input DataFrame : Output DataFrame : Below is the code for the same with explained steps : import org.apache.spark.SparkConf import org.apache.spark.sql.SparkSession import org.apache.spark.sql.functions._ object SplitArrayOfJsonsToRows {   def main(args: Array[String]) {     //Creating SparkSession     lazy val conf = new SparkConf().setAppName("split-array-of-json-to-row").set("spark.default.parallelism", "1")       .setIfMissing("spark.master", "local[*]")     lazy val sparkSession = SparkSession.builder()...
Read more