Senior Data Engineer

Consult Asia; Lucknow; On-site; INR 1 - 2; Full time

Job summary

Consult Asia in Lucknow, India, seeks an experienced Data Engineer (6–9 yrs) to design, build and migrate data pipelines using Spark and Python/Scala. You will move Hive and Control-M jobs to Airflow/DBT and integrate with Kafka, Solace, S3, and MinIO, deploying on Kubernetes for scalable production systems.

The role requires hands-on expertise with Airflow, DBT, Kafka and data lakehouse tools, with the ability to deliver independently and drive migrations end-to-end across the data stack.

Qualifications

• 6–9 years of experience in data engineering or related field.
• Strong hands-on expertise in Spark, Python/Scala and SQL.
• Proven ability to design, build and migrate data pipelines.

Responsibilities

• Develop and migrate data pipelines using Spark and Python/Scala.
• Move Hive and Control-M jobs to Airflow/DBT.
• Integrate with Kafka, Solace, S3, MinIO and deploy on Kubernetes for scalable, production-ready solutions.

Skills

Spark

Python/Scala

SQL

Data pipelines

Independent delivery

Migration experience

Tools

Airflow

DBT

Kafka

Kubernetes

Hive

Control-M

S3

MinIO

Solace

Job description

Required Candidate profile

Develop and migrate data pipelines using Spark, Python/Scala. Move Hive and Control-M jobs to Airflow/DBT. Integrate with Kafka, Solace, S3, MinIO, and deploy on Kubernetes ensuring scalable, automated, production-ready solutions.

Experienced Data Engineer (6–9 yrs) with strong Spark, Python/Scala, and SQL skills. Hands‑on in Airflow, DBT, Kafka, Kubernetes, and data lakehouse tools.Capable of independent delivery and migration