Senior Data Engineer

Target in India; Bengaluru; On-site; INR 2,500,000 - 4,000,000; Full time

Job summary

Target in India is seeking an experienced Senior Data Engineer to design, build, optimize, and support large-scale data platforms. You will work on complex data pipelines using Apache Spark, Scala, Hive, and Kafka, across batch and streaming workloads in a fast-paced setting.

The role requires hands-on expertise with GCP, BigQuery, and CI/CD practices, enabling cloud migrations and platform modernization. You will collaborate with multiple teams and own end-to-end data solutions.

Qualifications

• Hands-on expertise in Apache Spark and Scala for large-scale pipelines.
• Experience with high-volume batch and streaming workloads across sources.
• Strong experience with Hive and distributed data processing.
• Hands-on experience with Kafka and event-driven architectures.
• Programming in Scala and/or Python.
• Experience designing, developing, and troubleshooting large-scale ETL/data pipelines.
• Knowledge of data modeling, quality, performance tuning, and distributed systems.
• Experience with at least one cloud platform, preferably GCP.
• Experience with BigQuery and GCP services is preferred.
• Experience with ETL workflows and orchestration tools such as Oozie.
• Proficiency with Git, Jenkins, and CI/CD practices.

Responsibilities

• Design, develop, and optimize large-scale data pipelines using Spark and Scala.
• Work with high-volume batch and streaming workloads across multiple data sources.
• Develop and support data processing solutions using Hive, Kafka, and GCP services.
• Troubleshoot complex data, performance, and production issues with minimal supervision.
• Contribute to cloud migration and platform modernization initiatives.
• Build and maintain ETL workflows and data orchestration pipelines.
• Perform performance tuning and optimization of Spark jobs and datasets.
• Develop reusable, production-ready data solutions.
• Implement and maintain CI/CD pipelines and engineering automation.
• Collaborate with engineering, product, analytics, and platform teams to deliver end-to-end solutions.
• Participate in code reviews, testing, deployments, and operational support.
• Quickly understand existing systems and take ownership of deliverables.

Skills

Spark

Scala

Python

Hive

Kafka

GCP

BigQuery

ETL

CI/CD

Git

Jenkins

Tools

Oozie

Job description

We are looking for an experienced Senior Data Engineer who can quickly onboard and contribute to the design, development, optimization, and support of large-scale data platforms. The ideal candidate should be hands-on, comfortable working with complex data pipelines and distributed systems, and capable of independently troubleshooting and delivering solutions in a fast-paced environment with minimal handholding.

Responsibilities:

• Design, develop, and optimize large-scale data pipelines using Apache Spark and Scala.
• Work with high-volume batch and streaming workloads across multiple data sources.
• Develop and support data processing solutions using Hive, Kafka, and GCP services.
• Troubleshoot complex data, performance, and production issues with minimal supervision.
• Contribute to cloud migration and platform modernization initiatives.
• Build and maintain ETL workflows and data orchestration pipelines.
• Perform performance tuning and optimization of Spark jobs and large datasets.
• Develop reusable, maintainable, and production-ready data solutions.
• Implement and maintain CI/CD pipelines and engineering automation.
• Collaborate with engineering, product, analytics, and platform teams to deliver end-to-end solutions.
• Participate in code reviews, testing, production deployments, and operational support.
• Quickly understand existing systems and take ownership of assigned deliverables.

Requirements:

• Strong hands-on expertise in Apache Spark and Scala for building and optimizing large-scale data pipelines.
• Experience working with high-volume batch and streaming workloads across multiple data sources.
• Strong experience with Hive and distributed data processing.
• Hands-on experience with Kafka and event/streaming-based architectures.
• Strong programming experience in Scala and/or Python.
• Experience designing, developing, and troubleshooting large-scale ETL/data pipelines.
• Strong understanding of data modeling, data quality, performance tuning, and distributed systems.
• Experience working with at least one major cloud platform, GCP
• Experience with BigQuery and GCP services is preferred.
• Experience with ETL workflows and orchestration tools such as Oozie or equivalent.
• Hands-on experience with Git, Jenkins, and CI/CD practices, including automated build, testing, and deployment pipelines.
• Ability to independently troubleshoot complex data, performance, and production issues.

Preferred / Added Advantage:

• Strong Google Cloud Platform (GCP) experience.
• Exposure to BigQuery, Google Cloud Storage (GCS), Dataproc, and Pub/Sub.
• Experience migrating on-premises Spark/Hive workloads to cloud platforms.
• Experience with large-scale data migration and reconciliation.
• Knowledge of data quality and observability frameworks.
• Experience working with multiple upstream and downstream integrations.
• Exposure to multiple cloud environments.