Big Data Engineer

Kairos Technologies Inc.; Gurugram District, Chennai District; Hybrid; INR 2,500,000 - 4,000,000; Full time

Job summary

Kairos Technologies Inc. is seeking a skilled Big Data Engineer to design and optimize data pipelines on GCP, focusing on BigQuery, Dataproc, and Spark-based processing.

The ideal candidate has extensive hands-on experience with Python, PySpark, SQL, and data warehousing concepts. The role involves building scalable batch and streaming pipelines, managing data models, and leveraging Terraform for infrastructure as code on GCP.

Qualifications

• 57+ years of experience in Big Data Engineering or related roles.
• Strong programming skills in Python and PySpark.
• Hands-on expertise with GCP BigQuery, Dataproc, and related GCP services.
• Experience with distributed data processing frameworks (Spark, Hadoop).
• Strong SQL skills and experience optimizing complex BigQuery queries.
• Understanding of data warehousing, ETL/ELT pipelines, and data modeling concepts.
• Knowledge of Terraform for IaC on GCP.
• Implement batch and streaming data processing frameworks using PySpark and Spark SQL.
• BigQuery (data modeling, query optimization, partitioning/clustering)
• Dataproc (Spark/Hadoop cluster management, job orchestration)
• Cloud Storage
• Cloud Composer / Airflow

Skills

Python

PySpark

SQL

ETL/ELT design

Data modeling

Tools

GCP BigQuery

Dataproc

Cloud Storage

Terraform

Airflow

Spark/Hadoop

Job description

Description:

We are looking for a skilled Big Data Engineer

Mandatory Skills

are (GCP, BigQuery, Dataproc, Python, PySpark)

• 57+ years of experience in Big Data Engineering or related roles.


• Strong programming skills in Python and PySpark.


• Hands-on expertise with GCP BigQuery, Dataproc, and related GCP services.


• Experience with distributed data processing frameworks (Spark, Hadoop).


• Strong SQL skills and experience optimizing complex BigQuery queries.


• Understanding of data warehousing, ETL/ELT pipelines, and data modeling concepts.


• Knowledge of Terraform for IaC on GCP.


• Implement batch and streaming data processing frameworks using PySpark and Spark SQL.


• BigQuery (data modeling, query optimization, partitioning/clustering)


• Dataproc (Spark/Hadoop cluster management, job orchestration)


• Cloud Storage


• Cloud Composer / Airflow