Data Engineer - PySpark

Tekskills Inc.; Kolkata District; On-site; INR 1,800,000 - 2,600,000; Full time

Job summary

Tekskills Inc. invites a data engineering specialist to join our Kolkata-based team to design and optimize data pipelines using Spark, Hive, and Lakehouse technologies. You will work with Python/Scala/Java, orchestration tools like Airflow and Control-M, and DBT for transformations, building robust streaming and batch jobs.

Familiarity with Kafka, Solace, and object stores (S3/MinIO) is essential. Experience deploying with Docker/Kubernetes and working with Iceberg, Delta Lake, or Hudi will help

Qualifications

• Strong expertise in Apache Spark (batch + streaming) and Hive.
• Proficiency in Python, Scala, or Java.
• Knowledge of orchestration tools (Airflow / Control-M) and SQL transformation frameworks (DBT preferred).
• Experience working with Kafka, Solace, and object stores (S3, MinIO).
• Exposure to Docker/Kubernetes for deployment and data Lakehouse formats (Iceberg, Delta Lake, Hudi).

Responsibilities

• Strong expertise in Apache Spark (batch + streaming) and Hive.
• Proficiency in Python, Scala, or Java.
• Knowledge of orchestration tools (Airflow / Control-M) and SQL transformation frameworks (DBT preferred).
• Experience working with Kafka, Solace, and object stores (S3, MinIO).
• Exposure to Docker/Kubernetes for deployment.
• Hands on experience of data Lakehouse formats (Iceberg, Delta Lake, Hudi).

Skills

Python

Scala

Java

Spark

Hive

Tools

Apache Spark

Hive

Airflow

Control-M

DBT

Kafka

Solace

S3

MinIO

Docker

Kubernetes

Iceberg

Delta Lake

Hudi

Job description

Job Summary

Role & responsibilities Strong expertise in Apache Spark (batch + streaming) and Hive. Proficiency in Python, Scala, or Java. Knowledge of orchestration tools (Airflow / Control-M) and SQL transformation frameworks (DBT preferred). Experience working with Kafka, Solace, and object stores (S3, MinIO). Exposure to Docker/Kubernetes for deployment. Hands on experience of data Lakehouse formats (Iceberg, Delta Lake, Hudi).

Responsibilities

• Strong expertise in Apache Spark (batch + streaming) and Hive.
• Proficiency in Python, Scala, or Java.
• Knowledge of orchestration tools (Airflow / Control-M) and SQL transformation frameworks (DBT preferred).
• Experience working with Kafka, Solace, and object stores (S3, MinIO).
• Exposure to Docker/Kubernetes for deployment.
• Hands on experience of data Lakehouse formats (Iceberg, Delta Lake, Hudi).