How to Handle and Convert DateTime format in Spark-Scala.

Hi Friends,

In this post, I'd like to explain that how to handle and convert the date format in expected date format.
Below I'm providing few examples :

1. Change Date format to yyyy-mm-dd from dd-mm-yyyy or mm-dd-yyyy :

 Input Date : 03-04-2020T15:26:51+05:30

 Converted Output : 2020-04-03T15:26:051

Code to get the same output :

def changeDateFormat (inputDateString: String) = {
  var convertedDateFormat = ""  try {
    val inputDateFormat: SimpleDateFormat = new SimpleDateFormat("dd-MM-yyyy'T'HH:mm:sss")
    val date = inputDateFormat.parse(inputDateString)
    val df = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:sss")
    convertedDateFormat = df.format(date)
  } catch {
    case ex: Exception => {"Exception is : " + ex}
      logger.info("[HandleDateTimeUtils] Exception while converting time to 'yyyy-MM-dd'T'HH:mm:sss' Format. Input is: " + inputDateString, ex.getMessage)
      logger.error("[HandleDateTimeUtils] Exception while converting time to 'yyyy-MM-dd'T'HH:mm:sss' Format. Input is: " + inputDateString, ex.getMessage)
  }
  convertedDateFormat
}
Test this above method to handle a given timestamp :
val inputDateFormatToConvert: String = "03-04-2020T15:26:51+05:30"
println(changeDateFormat(inputDateFormatToConvert))     //2020-04-03T15:26:051

Output :




2. Convert given timestamp to Standard format :

 Input Date :  14-04-2020T15:26:51+05:30

 Converted Output :  2020-04-14 15:26:51

Code to get the same Output :
def getStandardDateFormat(inputDateString: String) = {
  val STANDARD_DATE_TIME_FORMAT = "yyyy-MM-dd HH:mm:ss"  var standardFormatDate = ""  try {
    val inputDate = new SimpleDateFormat("dd-MM-yyyy'T'HH:mm:sss").parse(inputDateString)
    val standardDateFormat = new SimpleDateFormat(STANDARD_DATE_TIME_FORMAT)
    standardFormatDate = standardDateFormat.format(inputDate)
  } catch {
    case ex: Exception =>
      logger.info("[HandleDateTimeUtils] Exception while converting time to Standard Date Format. Input is: " + inputDateString, ex.getMessage)
      logger.error("[HandleDateTimeUtils] Exception while converting time to Standard Date Format. Input is: " + inputDateString, ex.getMessage)
  }
  standardFormatDate
  
}
Test this above method to handle a given timestamp :
val inputDateFormatToStandardFormat: String = "14-04-2020T15:26:51+05:30"

println(getStandardDateFormat(inputDateFormatToStandardFormat))   //2020-04-14 15:26:51

 Output :

3. Convert Zulu timestamp to Standard format :

Input Date : 2020-04-03T20:17:46.384Z

Converted Output :  2020-04-03 20:17:46

Code to get the same Output :
def handleZuluTimeStamp(inputDateString: String) = {
  val STANDARD_DATE_TIME_FORMAT = "yyyy-MM-dd HH:mm:ss"  var zuluTime = ""  try {
    val inputDate = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.SSS'Z'").parse(inputDateString)
    val dateFormat = new SimpleDateFormat(STANDARD_DATE_TIME_FORMAT)
    zuluTime = dateFormat.format(inputDate)
  } catch {
    case ex: Exception =>
      logger.info("[HandleDateTimeUtils] Exception while converting time to Standard Date Format. Input is: " + inputDateString, ex.getMessage)
      logger.error("[HandleDateTimeUtils] Exception while converting time to Standard Date Format. Input is: " + inputDateString, ex.getMessage)
  }
  zuluTime
}

 Test this above method to handle a given timestamp :
val zuluTimeStamp = "2020-04-03T20:17:46.384Z"

println(handleZuluTimeStamp(zuluTimeStamp))     //2020-04-03 20:17:46
Output :


4. Convert timestamp to UTC format :

Input Date : 2017-08-01T14:45:10+05:30

Converted Output :  2017-08-01 09:15:10

Code to get the same Output :
def convertDateTimeToUTC(inputDateString: String) = {
  val STANDARD_DATE_TIME_FORMAT = "yyyy-MM-dd HH:mm:ss"  var utcTime = ""  try {
    val inputDate = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss").parse(inputDateString)
    val cal = Calendar.getInstance    // Creating Calendar    cal.setTime(inputDate)            // Sets Calendar Time/Date
    val timeZoneString = inputDateString.substring(inputDateString.length - 6, inputDateString.length)
    val hoursOfTimeZone = timeZoneString.substring(1, 3).toInt
    val minutesOfTimeZone = timeZoneString.substring(4, 6).toInt
    if (timeZoneString.contains("-")) {      //Add Hour and Minutes to Time      cal.add(Calendar.HOUR_OF_DAY, hoursOfTimeZone)
      cal.add(Calendar.MINUTE, minutesOfTimeZone)
    }
    else if (timeZoneString.contains("+")) {
      cal.add(Calendar.HOUR_OF_DAY, -hoursOfTimeZone)
      cal.add(Calendar.MINUTE, -minutesOfTimeZone)
    }
    val dateFormat = new SimpleDateFormat(STANDARD_DATE_TIME_FORMAT)
    utcTime = dateFormat.format(cal.getTime)
  } catch {
    case ex: Exception =>
      logger.info("[HandleDateTimeUtils] Exception while converting time to UTC. Input is: " + inputDateString, ex.getMessage)
      logger.error("[HandleDateTimeUtils] Exception while converting time to UTC. Input is: " + inputDateString, ex.getMessage)
  }
  utcTime
}

 Test this above method to handle a given timestamp :
val testTimeStampToUtc = "2017-08-01T14:45:10+05:30"

println(convertDateTimeToUTC1(testTimeStampToUtc))          //2017-08-01 09:15:10

 Output :

5. Convert Standard and Zulu timestamp to UTC format :

Standard Input TimeStamp : 2017-08-01T14:55:45+05:30
Zulu Input TimeStamp  : 2020-04-03T20:17:46.384Z

Converted Output for Standard TimeStamp :  2017-08-01 09:25:45
Converted Output for Zulu TimeStamp :  2020-04-03 20:17:46


Code to get the same Output :
def convertStandardAndZuluTimeStampToUTC(inputDateString: String) = {
  val STANDARD_DATE_TIME_FORMAT = "yyyy-MM-dd HH:mm:ss"  var utcTime = ""  try {
    val inputDate = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss").parse(inputDateString)
    val cal = Calendar.getInstance    // Creating Calendar    cal.setTime(inputDate)            // Sets Calendar Time/Date
    val timeZoneString = inputDateString.substring(inputDateString.length - 6, inputDateString.length)
    val hoursOfTimeZone = timeZoneString.substring(1, 3).toInt
    val minutesOfTimeZone = timeZoneString.substring(4, 6).toInt
    if (timeZoneString.contains("-")) {      //Add Hour and Minutes to Time      cal.add(Calendar.HOUR_OF_DAY, hoursOfTimeZone)
      cal.add(Calendar.MINUTE, minutesOfTimeZone)
    }
    else if (timeZoneString.contains("+")) {
      cal.add(Calendar.HOUR_OF_DAY, -hoursOfTimeZone)
      cal.add(Calendar.MINUTE, -minutesOfTimeZone)
    }
    val dateFormat = new SimpleDateFormat(STANDARD_DATE_TIME_FORMAT)
    utcTime = dateFormat.format(cal.getTime)
  } catch {
    case ex: NumberFormatException =>
      val zuluTime = handleZuluTimeStamp(inputDateString)
      if ("" == zuluTime) {
        logger.error("[HandleDateTimeUtils] NumberFormatException while parsing timezone to convert time to UTC. Input is: " + inputDateString, ex.getMessage)
      }
      else {
        logger.error("[HandleDateTimeUtils] input time " + inputDateString + " for the label : This is zulu time format")
        utcTime = zuluTime
      }
    case ex: Exception =>
      logger.info("[HandleDateTimeUtils] Exception while converting time to UTC. Input is: " + inputDateString, ex.getMessage)
      logger.error("[HandleDateTimeUtils] Exception while converting time to UTC. Input is: " + inputDateString, ex.getMessage)
      utcTime = ""  }
  utcTime
}
Test this above method to handle a given timestamp :
val convertStandardTimeStampToUtc = "2017-08-01T14:55:45+05:30" 
val convertZuluTimeStampToUtc = "2020-04-03T20:17:46.384Z"
println(convertStandardAndZuluTimeStampToUTC(convertStandardTimeStampToUtc))          //2017-08-01 09:25:45
println(convertStandardAndZuluTimeStampToUTC(convertZuluTimeStampToUtc))              //2020-04-03 20:17:46


Output :


Please find the below complete Scala Class for run and test above methods :

 import java.text.SimpleDateFormat
import java.util.Calendar
import org.slf4j.LoggerFactory

 /**
 * Created by anamika_singh3 on 04/13/2020.
 * This class contains methods to handle the timestamps and convert to desired format.
 */
object HandleDateTimeUtils extends App {

   val logger = LoggerFactory.getLogger(this.getClass)
  // 1. Change Date format to yyyy-mm-dd from dd-mm-yyyy or mm-dd-yyyy

   def changeDateFormat (inputDateString: String) = {
    var convertedDateFormat = ""
    try {
      val inputDateFormat: SimpleDateFormat = new SimpleDateFormat("dd-MM-yyyy'T'HH:mm:sss")
      val date = inputDateFormat.parse(inputDateString)
      val df = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:sss")
      convertedDateFormat = df.format(date)
    } catch {
      case ex: Exception => {"Exception is : " + ex}
        logger.info("[HandleDateTimeUtils] Exception while converting time to 'yyyy-MM-dd'T'HH:mm:sss' Format. Input is: " + inputDateString, ex.getMessage)
        logger.error("[HandleDateTimeUtils] Exception while converting time to 'yyyy-MM-dd'T'HH:mm:sss' Format. Input is: " + inputDateString, ex.getMessage)
    }
    convertedDateFormat
  }

   val inputDateFormatToConvert: String = "03-04-2020T15:26:51+05:30"
  println(changeDateFormat(inputDateFormatToConvert))     //2020-04-03T15:26:051
  // 2. Convert given timestamp to Standard format
  def getStandardDateFormat(inputDateString: String) = {
    val STANDARD_DATE_TIME_FORMAT = "yyyy-MM-dd HH:mm:ss"
    var standardFormatDate = ""
    try {
      val inputDate = new SimpleDateFormat("dd-MM-yyyy'T'HH:mm:sss").parse(inputDateString)
      val standardDateFormat = new SimpleDateFormat(STANDARD_DATE_TIME_FORMAT)
      standardFormatDate = standardDateFormat.format(inputDate)
    } catch {
      case ex: Exception =>
        logger.info("[HandleDateTimeUtils] Exception while converting time to Standard Date Format. Input is: " + inputDateString, ex.getMessage)
        logger.error("[HandleDateTimeUtils] Exception while converting time to Standard Date Format. Input is: " + inputDateString, ex.getMessage)
    }
    standardFormatDate

   }

   val inputDateFormatToStandardFormat: String = "14-04-2020T15:26:51+05:30"
  println(getStandardDateFormat(inputDateFormatToStandardFormat))   //2020-04-14 15:26:51

   // 3. Convert Zulu timestamp to Standard format :
  def handleZuluTimeStamp(inputDateString: String) = {
    val STANDARD_DATE_TIME_FORMAT = "yyyy-MM-dd HH:mm:ss"
    var zuluTime = ""
    try {
      val inputDate = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.SSS'Z'").parse(inputDateString)
      val dateFormat = new SimpleDateFormat(STANDARD_DATE_TIME_FORMAT)
      zuluTime = dateFormat.format(inputDate)
    } catch {
      case ex: Exception =>
        logger.info("[HandleDateTimeUtils] Exception while converting time to Standard Date Format. Input is: " + inputDateString, ex.getMessage)
        logger.error("[HandleDateTimeUtils] Exception while converting time to Standard Date Format. Input is: " + inputDateString, ex.getMessage)
    }
    zuluTime
  }
  val zuluTimeStamp = "2020-04-03T20:17:46.384Z"
  println(handleZuluTimeStamp(zuluTimeStamp))     //2020-04-03 20:17:46

   // 4. Convert timestamp to UTC format :
  def convertDateTimeToUTC(inputDateString: String) = {
    val STANDARD_DATE_TIME_FORMAT = "yyyy-MM-dd HH:mm:ss"
    var utcTime = ""
    try {
      val inputDate = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss").parse(inputDateString)
      val cal = Calendar.getInstance    // Creating Calendar
      cal.setTime(inputDate)            // Sets Calendar Time/Date

       val timeZoneString = inputDateString.substring(inputDateString.length - 6, inputDateString.length)
      val hoursOfTimeZone = timeZoneString.substring(1, 3).toInt
      val minutesOfTimeZone = timeZoneString.substring(4, 6).toInt
      if (timeZoneString.contains("-")) {      //Add Hour and Minutes to Time
        cal.add(Calendar.HOUR_OF_DAY, hoursOfTimeZone)
        cal.add(Calendar.MINUTE, minutesOfTimeZone)
      }
      else if (timeZoneString.contains("+")) {
        cal.add(Calendar.HOUR_OF_DAY, -hoursOfTimeZone)
        cal.add(Calendar.MINUTE, -minutesOfTimeZone)
      }
      val dateFormat = new SimpleDateFormat(STANDARD_DATE_TIME_FORMAT)
      utcTime = dateFormat.format(cal.getTime)
    } catch {
      case ex: Exception =>
        logger.info("[HandleDateTimeUtils] Exception while converting time to UTC. Input is: " + inputDateString, ex.getMessage)
        logger.error("[HandleDateTimeUtils] Exception while converting time to UTC. Input is: " + inputDateString, ex.getMessage)
    }
    utcTime
  }

   //val testTimeStampToUtc = "2017-08-01T14:45:10+05:30"
  //println(convertDateTimeToUTC1(testTimeStampToUtc))          //2017-08-01 09:15:10

   // 5. Convert Standard and Zulu timestamp to UTC format :
  def convertStandardAndZuluTimeStampToUTC(inputDateString: String) = {
    val STANDARD_DATE_TIME_FORMAT = "yyyy-MM-dd HH:mm:ss"
    var utcTime = ""
    try {
      val inputDate = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss").parse(inputDateString)
      val cal = Calendar.getInstance    // Creating Calendar
      cal.setTime(inputDate)            // Sets Calendar Time/Date

       val timeZoneString = inputDateString.substring(inputDateString.length - 6, inputDateString.length)
      val hoursOfTimeZone = timeZoneString.substring(1, 3).toInt
      val minutesOfTimeZone = timeZoneString.substring(4, 6).toInt
      if (timeZoneString.contains("-")) {      //Add Hour and Minutes to Time
        cal.add(Calendar.HOUR_OF_DAY, hoursOfTimeZone)
        cal.add(Calendar.MINUTE, minutesOfTimeZone)
      }
      else if (timeZoneString.contains("+")) {
        cal.add(Calendar.HOUR_OF_DAY, -hoursOfTimeZone)
        cal.add(Calendar.MINUTE, -minutesOfTimeZone)
      }
      val dateFormat = new SimpleDateFormat(STANDARD_DATE_TIME_FORMAT)
      utcTime = dateFormat.format(cal.getTime)
    } catch {
      case ex: NumberFormatException =>
        val zuluTime = handleZuluTimeStamp(inputDateString)
        if ("" == zuluTime) {
          logger.error("[HandleDateTimeUtils] NumberFormatException while parsing timezone to convert time to UTC. Input is: " + inputDateString, ex.getMessage)
        }
        else {
          logger.error("[HandleDateTimeUtils] input time " + inputDateString + " for the label : This is zulu time format")
          utcTime = zuluTime
        }
      case ex: Exception =>
        logger.info("[HandleDateTimeUtils] Exception while converting time to UTC. Input is: " + inputDateString, ex.getMessage)
        logger.error("[HandleDateTimeUtils] Exception while converting time to UTC. Input is: " + inputDateString, ex.getMessage)
        utcTime = ""
    }
    utcTime
  }

    val convertStandardTimeStampToUtc = "2017-08-01T14:55:45+05:30"
   val convertZuluTimeStampToUtc = "2020-04-03T20:17:46.384Z"
  println(convertStandardAndZuluTimeStampToUTC(convertStandardTimeStampToUtc))          //2017-08-01 09:25:45
  println(convertStandardAndZuluTimeStampToUTC(convertZuluTimeStampToUtc))              //2020-04-03 20:17:46

 }

I Hope, This Post was helpful, please do like, comment and share.
Thank You !

Comments

Post a Comment

Popular posts from this blog

Transformations and Actions in Spark

Image
Hello Friends, Today I’ll try to explain about Transformations and Actions in Spark. We know that Basically, Spark  RDD supports two types of Operations :          ·    Transformations          ·    Actions Transformations : Transformations are kind of operations which will transform our RDD data from one form to another form, and when we apply this operation on any RDD, we get a new RDD with transformed data   as RDDs in Spark are immutable. Operations like map, flatMap, filter, join, union are transformations. Now there is a point to be noted here and that is when we apply the transformation on any RDD it will not perform the operation immediately. It creates a DAG(Directed Acyclic Graph) using the applied operations, source RDD and functions used for transformation. And it keeps on building this graph using the references till we apply any action operation on the last lined up RDD. That is why the transformation in Spark are lazy. After the transformatio
Read more

How to convert XML String to JSON String in Spark-Scala

Image
Hello Friends, Today I'd like to share the method for conversion of XML data to Json data using Scala. This is done with org.json library which we use  to encode or decode  JSON  data. Input XML : <Summary_Log> <Monitor>        <HardwareInfo>FD2F</HardwareInfo>       <Id>0</Id>       <Monitor_Type>LCD</Monitor_Type>       <Vendor_Data>110HL</Vendor_Data> </Monitor> <Processor>       <HardwareInfo>FD2F</HardwareInfo>      <Id>1</Id>        <Used>80</Used>      <Utilization>19</Utilization> </Processor> </Summary_Log> Output Json : {"Summary_Log": {     "Monitor": {         "Monitor_Type": "LCD",         "Vendor_Data": "110HL",         "HardwareInfo": "FD2F",         "Id": 0     },     "Processor": {        
Read more

How to Convert a Spark DataFrame to Map in Scala

Image
Hi Friends, In this post, I am going to explain that how we can convert Spark DataFrame to Map. Input DataFrame : Output map : Below is the explained code with all the steps along with its Output.   import org.apache.spark.SparkConf import org.apache.spark.sql.{Row, SparkSession} import org.apache.spark.sql.types.{IntegerType, StringType, StructField, StructType} object ConvertDFToMap {   def main(args: Array[String]) {     //Creating SparkSession     lazy val conf = new SparkConf().setAppName("DataFrame-to-Map").set("spark.default.parallelism", "1")       .setIfMissing("spark.master", "local[*]")     lazy val sparkSession = SparkSession.builder().config(conf).getOrCreate()     // Creating Seq of Raw Data to create DataFrame to test     val rawData = Seq(       Row(1, "Anamika", "Uttar Pradesh"),       Row(2, "Ananya", "Delhi"),       Row(3, "Ambika",
Read more