CICD for Data Engineers with easy understanding!

Let's say we are working on a Customer Analysis Project & have a Jira ticket assigned to us as CA-111

If we are a developer, we would create a feature branch as below :
feature/CA-111 and work on it.

As soon as we make a git push, and GitHub receives the new code, a pipeline should run which involves below steps :

1. Build : Creating a virtual environment & install all the dependencies (In case of python).
2. Test : Run unit test cases / Quality checks.
3. Package : Create a package, can be a zip of code.
4. Deploy : Send the code bundle to edge node using SCP (Secure Copy).

If we do all of these manually then it will be a time consuming and error prone. So all of the above steps should run as a automated pipeline step by step. We can automate it using a automation server such as Jenkins.
So anytime a new branch is created, or a git push happens, all of the build -> test -> package -> deploy should run without any manual involvement.

Now let us understand the branching structure based on different environments which most of the data engineering projects follow.

Feature Branches -> Dev -> Test -> UAT -> main/Master (Prod)

Feature Branch -> This should be the branch where developers should code their new changes and after testing the code properly, we can merge the code with Development and other branches as below :
Dev -> Development Branch
Test -> For QA team to test
UAT -> For user acceptance testing
main/Master -> This is the production branch.

Feature branches are short lived that means once feature branch is merge to a higher branch, we can delete it. All other branches will be there for always. So, when we make a code push to Feature Branch in GitHub then all the 4 steps in CICD pipeline will run.

For the Feature Branch, if we see that all the 4 steps ran fine, then we can raise a pull request to merge our code to Dev Branch. once the reviewers see and merges it, the automated pipeline should run again on Dev Branch.

Same way when code is merged to Test, UAT & main/Master the pipeline should run.

Most of the companies follow the similar structure for CICD in data engineering projects.

As a Data Engineer we should not go too deep into it, but we should have a fair idea. Ideally the entire process would be set, we should have to just follow that.


I tried my best to explain the CICD process in simple words and I hope you would have liked the post. If you like then please Like, Comment and Share.

Thank You!

Comments

Post a Comment

Popular posts from this blog

Transformations and Actions in Spark

Image
Hello Friends, Today I’ll try to explain about Transformations and Actions in Spark. We know that Basically, Spark  RDD supports two types of Operations :          ·    Transformations          ·    Actions Transformations : Transformations are kind of operations which will transform our RDD data from one form to another form, and when we apply this operation on any RDD, we get a new RDD with transformed data   as RDDs in Spark are immutable. Operations like map, flatMap, filter, join, union are transformations. Now there is a point to be noted here and that is when we apply the transformation on any RDD it will not perform the operation immediately. It creates a DAG(Directed Acyclic Graph) using the applied operations, source RDD and functions used for transformation. And it keeps on building this graph using the references till we apply any action operation on the last lined up RDD. That is why the transformation in Spark are lazy. After the transformatio
Read more

How to Convert a Spark DataFrame to Map in Scala

Image
Hi Friends, In this post, I am going to explain that how we can convert Spark DataFrame to Map. Input DataFrame : Output map : Below is the explained code with all the steps along with its Output.   import org.apache.spark.SparkConf import org.apache.spark.sql.{Row, SparkSession} import org.apache.spark.sql.types.{IntegerType, StringType, StructField, StructType} object ConvertDFToMap {   def main(args: Array[String]) {     //Creating SparkSession     lazy val conf = new SparkConf().setAppName("DataFrame-to-Map").set("spark.default.parallelism", "1")       .setIfMissing("spark.master", "local[*]")     lazy val sparkSession = SparkSession.builder().config(conf).getOrCreate()     // Creating Seq of Raw Data to create DataFrame to test     val rawData = Seq(       Row(1, "Anamika", "Uttar Pradesh"),       Row(2, "Ananya", "Delhi"),       Row(3, "Ambika",
Read more

How to Handle and Convert DateTime format in Spark-Scala.

Image
Hi Friends, In this post, I'd like to explain that how to handle and convert the date format in expected date format. Below I'm providing few examples : 1. Change Date format to yyyy-mm-dd from dd-mm-yyyy or mm-dd-yyyy : Input Date :  03-04-2020T15:26:51+05:30 Converted Output : 2020-04-03T15:26:051 Code to get the same output : def changeDateFormat (inputDateString: String ) = { var convertedDateFormat = "" try { val inputDateFormat: SimpleDateFormat = new SimpleDateFormat( "dd-MM-yyyy'T'HH:mm:sss" ) val date = inputDateFormat.parse(inputDateString) val df = new SimpleDateFormat( "yyyy-MM-dd'T'HH:mm:sss" ) convertedDateFormat = df.format(date) } catch { case ex: Exception => { "Exception is : " + ex } logger .info( "[HandleDateTimeUtils] Exception while converting time to 'yyyy-MM-dd'T'HH:mm:sss' Format. Input is: " + inputDateString , ex.getMessag
Read more