Data Engineer - PySpark

Tekskills; Chennai District; On-site; INR 1,500,000 - 2,100,000; Full time

Job summary

Tekskills in Chennai is seeking a data engineering specialist skilled in Apache Spark (batch + streaming) and Hive, with strong Python, Scala or Java abilities. The role requires familiarity with orchestration tools (Airflow or Control-M) and DBT, plus experience with Kafka, Solace and object stores (S3, MinIO).

Docker/Kubernetes experience and knowledge of lakehouse formats (Iceberg, Delta Lake, Hudi) are a plus.

Qualifications

• Experience with Spark (batch and streaming) and Hive.
• Proficiency in Python, Scala or Java.
• Knowledge of orchestration tools (Airflow / Control-M) and DBT for SQL transformations.
• Experience with Kafka, Solace and object stores (S3, MinIO).
• Exposure to Docker/Kubernetes for deployment and data lakehouse formats.

Responsibilities

• Strong expertise in Spark (batch + streaming) and Hive.
• Proficiency in Python, Scala, or Java.
• Knowledge of orchestration tools (Airflow / Control-M) and DBT for SQL transformations.
• Experience with Kafka, Solace and object stores (S3, MinIO).
• Exposure to Docker/Kubernetes for deployment.
• Hands on experience with data lakehouse formats (Iceberg, Delta Lake, Hudi).

Skills

Spark (batch & streaming)

Hive

Python/Scala/Java

Orchestration tools

SQL transformation frameworks

Kafka

Solace

Cloud object stores

Docker/Kubernetes

Lakehouse formats

Tools

Airflow

Control-M

DBT

S3

MinIO

Docker

Kubernetes

Iceberg

Delta Lake

Hudi

Job description

Job Summary

Role & responsibilities Strong expertise in Apache Spark (batch + streaming) and Hive. Proficiency in Python, Scala, or Java. Knowledge of orchestration tools (Airflow / Control-M) and SQL transformation frameworks (DBT preferred). Experience working with Kafka, Solace, and object stores (S3, MinIO). Exposure to Docker/Kubernetes for deployment. Hands on experience of data Lakehouse formats (Iceberg, Delta Lake, Hudi).

Responsibilities

• Strong expertise in Apache Spark (batch + streaming) and Hive.
• Proficiency in Python, Scala, or Java.
• Knowledge of orchestration tools (Airflow / Control-M) and SQL transformation frameworks (DBT preferred).
• Experience working with Kafka, Solace, and object stores (S3, MinIO).
• Exposure to Docker/Kubernetes for deployment.
• Hands on experience of data Lakehouse formats (Iceberg, Delta Lake, Hudi).