Senior Data Engineer (Spark, Scala) | 9 to 13 YoE | US-based large retail organisation | Bengaluru | Immediate/Early joiners

Geektrust; Bengaluru; Presencial; INR 2.500.000 - 4.500.000; Jornada completa

Descripción de la vacante

Geektrust in Bengaluru seeks a Senior Data Engineer with 7–12 years of experience to design, develop and optimise large-scale data pipelines using Apache Spark and Scala across batch and streaming workloads.

You will build solutions with Hive, Kafka, BigQuery, Dataproc and GCS, craft ETL workflows with shell scripting, and drive cloud migration and data platform modernisation in a fast-paced, collaborative environment.

Formación

• 7–12 years of relevant experience in Data Engineering with strong hands-on expertise in Apache Spark and Scala.
• Strong experience with GCP services, particularly Dataproc, BigQuery and Google Cloud Storage (GCS).
• Strong SQL skills, particularly BigQuery SQL, for data transformation, validation and reconciliation.
• Hands-on experience with large-scale ETL pipelines, distributed data processing, Hive, Kafka and high-volume batch/streaming workloads.
• Strong Unix/Linux shell scripting skills for automation, data movement and pipeline orchestration, along with familiarity with Git, Jenkins and CI/CD.
• Strong understanding of data modelling, data quality, distributed systems and performance optimisation, with the ability to independently troubleshoot complex issues and deliver solutions with minimal handholding.

Responsabilidades

• Design, develop and optimise large-scale data pipelines using Apache Spark and Scala across high-volume batch and streaming workloads.
• Build and support data processing solutions using Hive, Kafka, BigQuery, Dataproc and GCS across multiple data sources.
• Develop, maintain and optimise ETL workflows, data orchestration pipelines and engineering automation using shell scripting and relevant tools.
• Troubleshoot complex data, performance and production issues; perform Spark job tuning and optimise processing across large datasets.
• Drive cloud migration, data migration and platform modernisation initiatives, including workload migration, data reconciliation and validation.
• Take end-to-end ownership of assigned deliverables, from understanding existing systems and solution development through testing, code reviews, CI/CD deployments and production support, collaborating with cross-functional teams.

Conocimientos

Apache Spark

Scala

Unix/Linux shell scripting

CI/CD

Data modelling

Data quality

Performance optimisation

Troubleshooting

Cloud migration

Herramientas

Hive

Kafka

BigQuery

Dataproc

GCS

Spark

Scala

Git

Jenkins

Descripción del empleo

• 7–12 years of relevant experience in Data Engineering, with strong hands-on expertise in Apache Spark and Scala.
• Strong experience with GCP services, particularly Dataproc, BigQuery and Google Cloud Storage (GCS).
• Strong SQL skills, particularly BigQuery SQL, for data transformation, validation and reconciliation.
• Hands-on experience with large-scale ETL pipelines, distributed data processing, Hive, Kafka and high-volume batch/streaming workloads.
• Strong Unix/Linux shell scripting skills for automation, data movement and pipeline orchestration, along with familiarity with Git, Jenkins and CI/CD.
• Strong understanding of data modelling, data quality, distributed systems and performance optimisation, with the ability to independently troubleshoot complex issues and deliver solutions with minimal handholding.

What you would do:

• Design, develop and optimise large-scale data pipelines using Apache Spark and Scala across high-volume batch and streaming workloads.
• Build and support data processing solutions using Hive, Kafka, BigQuery, Dataproc and GCS across multiple data sources.
• Develop, maintain and optimise ETL workflows, data orchestration pipelines and engineering automation using shell scripting and relevant tools.
• Troubleshoot complex data, performance and production issues; perform Spark job tuning and optimise processing across large datasets.
• Drive cloud migration, data migration and platform modernisation initiatives, including workload migration, data reconciliation and validation.
• Take end-to-end ownership of assigned deliverables, from understanding existing systems and solution development through testing, code reviews, CI/CD deployments and production support, collaborating with cross-functional teams.