Senior Data Engineer

Consult Asia; Bengaluru; On-site; INR 2,500,000 - 4,000,000; Full time

Job summary

Consult Asia is seeking an Experienced Data Engineer to design and migrate scalable data pipelines. The candidate will leverage Spark, Python and SQL to move Hive and Control-M workflows to Airflow and DBT, while integrating Kafka, Solace and cloud storage like S3/MinIO.

Kubernetes deployment ensures production-grade reliability. The ideal candidate has 6–9 years of hands-on data engineering, with proficiency in Airflow, DBT, Kafka, Kubernetes and data lakehouse tools, and can deliver

Qualifications

• 6–9 years of data engineering experience.
• Strong Spark, Python/Scala, and SQL skills.
• Hands-on with Airflow, DBT, Kafka, Kubernetes and data lakehouse tools.

Responsibilities

• Develop and migrate data pipelines using Spark, Python/Scala.
• Migrate Hive and Control-M jobs to Airflow/DBT.
• Integrate with Kafka, Solace, S3/MinIO and deploy on Kubernetes for scalable solutions.

Skills

Spark

Python

SQL

Scala

Tools

Airflow

DBT

Kafka

Kubernetes

Hive

S3

MinIO

Control-M

Job description

Required Candidate profile

Develop and migrate data pipelines using Spark, Python/Scala. Move Hive and Control-M jobs to Airflow/DBT. Integrate with Kafka, Solace, S3, MinIO, and deploy on Kubernetes ensuring scalable, automated, production-ready solutions.

Experienced Data Engineer (6–9 yrs) with strong Spark, Python/Scala, and SQL skills. Hands‑on in Airflow, DBT, Kafka, Kubernetes, and data lakehouse tools.Capable of independent delivery and migration