Senior Data Engineer

Consult Asia; Hyderabad; Sur place; INR 1 600 000 - 2 100 000; Plein temps

Résumé du poste

Consult Asia is seeking a seasoned Data Engineer to develop and migrate data pipelines using Spark and Python/Scala. You will move Hive and Control-M jobs to Airflow/DBT, and integrate with Kafka, Solace, S3, MinIO while deploying on Kubernetes to ensure scalable, automated, production-ready solutions.

The role requires 6–9 years of hands-on experience with Spark, Python/Scala, SQL, Airflow, DBT, Kafka, and Kubernetes, along with data lakehouse tooling and independent delivery capability.

Qualifications

• 6–9 years of experience as a Data Engineer.
• Strong Spark, Python/Scala, and SQL skills.
• Hands-on experience with Airflow, DBT, Kafka, Kubernetes, and data lakehouse tooling.

Responsabilités

• Develop and migrate data pipelines using Spark, Python/Scala.
• Migrate Hive and Control-M jobs to Airflow/DBT.
• Integrate with Kafka, Solace, S3, MinIO and deploy on Kubernetes for scalable, automated, production-ready solutions.

Connaissances

Spark

Python/Scala

SQL

Airflow

DBT

Kafka

Kubernetes

Data Lakehouse

Outils

Hive

Control-M

S3

MinIO

Description du poste

Required Candidate profile

Develop and migrate data pipelines using Spark, Python/Scala. Move Hive and Control-M jobs to Airflow/DBT. Integrate with Kafka, Solace, S3, MinIO, and deploy on Kubernetes ensuring scalable, automated, production-ready solutions.

Experienced Data Engineer (6–9 yrs) with strong Spark, Python/Scala, and SQL skills. Hands‑on in Airflow, DBT, Kafka, Kubernetes, and data lakehouse tools.Capable of independent delivery and migration