Senior Data Engineer

Consult Asia; Nagpur District; On-site; INR 2,500,000 - 4,000,000; Full time

Job summary

Consult Asia is seeking an experienced Data Engineer to design and migrate end-to-end data pipelines. You will leverage Spark and Python/Scala, with strong SQL, to build reliable data flows and support data lakehouse initiatives.

You will move Hive and Control-M workloads to Airflow/DBT, integrate with Kafka, Solace, S3, and MinIO, and deploy on Kubernetes to ensure scalable, production-ready solutions across environments.

Qualifications

• 6–9 yrs experience in data engineering with Spark, Python/Scala and SQL.
• Hands-on experience with Airflow, DBT, Kafka, Kubernetes.
• Experience with data lakehouse tools and migration projects.

Responsibilities

• Develop and migrate data pipelines using Spark, Python/Scala and SQL.
• Migrate Hive and Control-M jobs to Airflow/DBT.
• Integrate with Kafka, Solace, S3, MinIO and deploy on Kubernetes for scalable, production-ready solutions.

Skills

Spark

Python/Scala

SQL

Tools

Airflow

DBT

Kafka

Kubernetes

Hive

Control-M

S3

MinIO

Job description

Develop and migrate data pipelines using Spark, Python/Scala. Move Hive and Control-M jobs to Airflow/DBT. Integrate with Kafka, Solace, S3, MinIO, and deploy on Kubernetes ensuring scalable, automated, production-ready solutions.

Required Candidate profile

Experienced Data Engineer (6–9 yrs) with strong Spark, Python/Scala, and SQL skills. Hands-on in Airflow, DBT, Kafka, Kubernetes, and data lakehouse tools.Capable of independent delivery and migration