Data Engineer - PySpark

Tekskills Inc.; Chennai; On-site; INR 1,500,000 - 2,100,000; Full time

Job summary

Tekskills Inc. is seeking a data engineering professional with deep expertise in Apache Spark (batch and streaming) and Hive, plus strong coding in Python, Scala, or Java.

You should know orchestration tools like Airflow or Control-M and DBT for SQL transformations. Experience with Kafka, Solace and S3/MinIO, plus Docker/Kubernetes for deployment, and lakehouse formats Iceberg, Delta Lake and Hudi is essential.

Qualifications

• Proficiency in Spark (batch + streaming) and Hive.
• Strong programming skills in Python, Scala or Java.
• Experience with orchestration tools (Airflow/Control-M) and SQL transformation (DBT preferred).
• Experience with Kafka, Solace, and object stores such as S3/MinIO.
• Exposure to containerization and deployment with Docker/Kubernetes.
• Hands-on experience with data lakehouse formats (Iceberg, Delta Lake, Hudi).

Responsibilities

• Strong expertise in Apache Spark (batch + streaming) and Hive.
• Proficiency in Python, Scala, or Java.
• Knowledge of orchestration tools (Airflow / Control-M) and DBT
• Experience working with Kafka, Solace, and object stores (S3, MinIO).
• Exposure to Docker/Kubernetes for deployment.
• Hands on experience of data Lakehouse formats (Iceberg, Delta Lake, Hudi).

Skills

Apache Spark

Hive

Python

Scala

Java

Airflow

Control-M

DBT

Kafka

Solace

S3/MinIO

Docker/Kubernetes

Iceberg

Delta Lake

Hudi

Tools

DBT

Airflow

Kafka

Solace

Docker

Kubernetes

Job description

Job Summary

Role & responsibilities Strong expertise in Apache Spark (batch + streaming) and Hive. Proficiency in Python, Scala, or Java. Knowledge of orchestration tools (Airflow / Control-M) and SQL transformation frameworks (DBT preferred). Experience working with Kafka, Solace, and object stores (S3, MinIO). Exposure to Docker/Kubernetes for deployment. Hands on experience of data Lakehouse formats (Iceberg, Delta Lake, Hudi).

Responsibilities

• Strong expertise in Apache Spark (batch + streaming) and Hive.
• Proficiency in Python, Scala, or Java.
• Knowledge of orchestration tools (Airflow / Control-M) and SQL transformation frameworks (DBT preferred).
• Experience working with Kafka, Solace, and object stores (S3, MinIO).
• Exposure to Docker/Kubernetes for deployment.
• Hands on experience of data Lakehouse formats (Iceberg, Delta Lake, Hudi).