Data Engineer - PySpark

Tekskills; Bengaluru; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

Tekskills seeks a skilled data engineer with deep Apache Spark expertise (batch and streaming) and Hive mastery. Proficiency in Python, Scala, or Java is essential.

The role requires familiarity with Airflow or Control-M, DBT, and experience with Kafka, Solace, and S3/MinIO for scalable data pipelines. Candidate should be comfortable deploying with Docker/Kubernetes and working with Lakehouse formats like Iceberg, Delta Lake, or Hudi.

Qualifications

• Proficient in Python, Scala or Java.
• Strong knowledge of Spark (batch and streaming) and Hive.
• Experience with orchestration tools (Airflow, Control-M).
• Experience with SQL transformation frameworks (DBT preferred).
• Experience with Kafka, Solace and object stores (S3/MinIO).
• Exposure to Docker and Kubernetes for deployment.
• Hands-on experience with data Lakehouse formats (Iceberg, Delta Lake, Hudi).

Responsibilities

• Strong expertise in Apache Spark (batch + streaming) and Hive.
• Proficiency in Python, Scala, or Java.
• Knowledge of orchestration tools (Airflow / Control-M) and SQL transformation frameworks (DBT preferred).
• Experience working with Kafka, Solace, and object stores (S3, MinIO).
• Exposure to Docker/Kubernetes for deployment.
• Hands on experience of data Lakehouse formats (Iceberg, Delta Lake, Hudi).

Skills

Python

Scala

Java

Tools

Apache Spark

Hive

Airflow

Control-M

DBT

Kafka

Solace

Docker

Kubernetes

Iceberg

Delta Lake

Hudi

S3

MinIO

Job description

Job Summary

Role & responsibilities Strong expertise in Apache Spark (batch + streaming) and Hive. Proficiency in Python, Scala, or Java. Knowledge of orchestration tools (Airflow / Control-M) and SQL transformation frameworks (DBT preferred). Experience working with Kafka, Solace, and object stores (S3, MinIO). Exposure to Docker/Kubernetes for deployment. Hands on experience of data Lakehouse formats (Iceberg, Delta Lake, Hudi).

Responsibilities

• Strong expertise in Apache Spark (batch + streaming) and Hive.
• Proficiency in Python, Scala, or Java.
• Knowledge of orchestration tools (Airflow / Control-M) and SQL transformation frameworks (DBT preferred).
• Experience working with Kafka, Solace, and object stores (S3, MinIO).
• Exposure to Docker/Kubernetes for deployment.
• Hands on experience of data Lakehouse formats (Iceberg, Delta Lake, Hudi).