Senior Data Engineer

Consult Asia; Mumbai; On-site; INR 3,500,000 - 6,000,000; Full time

Job summary

Consult Asia is seeking an experienced Data Engineer to develop and migrate data pipelines using Spark and Python/Scala. You will move Hive and Control-M jobs to Airflow/DBT and integrate with Kafka, Solace, S3, MinIO, deploying on Kubernetes for scalable, automated, production-ready solutions.

The role requires 6–9 years of hands-on experience with Spark, Python/Scala, SQL, Airflow, DBT, Kafka and Kubernetes, with a track record of independent delivery and migration.

Qualifications

• 6–9 years of experience as a data engineer with Spark, Python/Scala and SQL
• Hands-on experience with Airflow, DBT, Kafka, Kubernetes
• Experience migrating data pipelines and moving Hive/Control-M jobs to Airflow/DBT

Responsibilities

• Develop and migrate data pipelines using Spark and Python/Scala
• Move Hive and Control-M jobs to Airflow/DBT
• Integrate with Kafka, Solace, S3/MinIO and deploy on Kubernetes for scalable, production-ready solutions

Skills

Spark

Python

Scala

SQL

Tools

Airflow

DBT

Kafka

Kubernetes

Hive

Control-M

Job description

Required Candidate profile

Develop and migrate data pipelines using Spark, Python/Scala. Move Hive and Control-M jobs to Airflow/DBT. Integrate with Kafka, Solace, S3, MinIO, and deploy on Kubernetes ensuring scalable, automated, production-ready solutions.

Experienced Data Engineer (6–9 yrs) with strong Spark, Python/Scala, and SQL skills. Hands‑on in Airflow, DBT, Kafka, Kubernetes, and data lakehouse tools.Capable of independent delivery and migration