PySpark Engineer

GIRI CONSULTANCY; Hyderabad; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

GIRI CONSULTANCY in Hyderabad seeks a PySpark Engineer to design, build, and optimise Spark applications and ETL data pipelines for a complex data lake. You will work on batch and streaming workloads, Delta Lake implementations, and CI/CD automation for PySpark.

You will lead a small technical team, size clusters across Spark Standalone, YARN, and Kubernetes, and collaborate with agile teams to implement data strategies and dataflows in a high-availability environment.

Qualifications

• Experience designing and developing Spark applications using PySpark.
• Hands-on experience building and maintaining ETL data pipelines.
• Expertise in Python development.
• Proficiency in writing SQL scripts.
• Spark Streaming and Structured Streaming knowledge is mandatory.
• Experience with Delta Lake for enterprise-grade implementation.
• Experience with cluster managers: Spark Standalone, YARN, Kubernetes.
• Knowledge of Big Data on Cloud, preferably GCP Dataproc and GCS.
• Strong communication and time management skills.
• Ability to manage a small team as technical specialists.

Responsibilities

• Design and develop Spark applications using PySpark for enterprise-scale data processing.
• Build and maintain ETL pipelines feeding a complex data lake.
• Implement Delta Lake for reliable data storage and access.
• Perform optimisation and performance tuning of Spark applications.
• Develop Spark Streaming workloads and enable CI/CD for PySpark deployments.
• Size Spark clusters and manage resources across cluster managers.
• Set up and operate highly available Spark clusters with monitoring.
• Collaborate with agile teams to implement data strategies and dataflows.
• Forecast environment requirements based on demand from multiple teams.
• Provide technical guidance and lead a small team of specialists.

Skills

PySpark

ETL pipelines

Python

SQL

Delta Lake

Spark Streaming

CI/CD

Cluster sizing

GCP Dataproc

Communication

Team leadership

Tools

Kubernetes

YARN

Spark Standalone

Airflow

GCS

Job description

Job Description:

Role Summary

We are seeking a PySpark Engineer to design, build, and optimise Spark applications and ETL data pipelines for a complex enterprise data lake. The role covers batch and streaming workloads, Delta Lake implementation, Spark performance tuning and cluster sizing, and CI/CD automation for PySpark. The engineer will operate highly available Spark clusters with monitoring, work with Agile application development teams on data strategy and dataflows, and act as a technical specialist with responsibility for guiding a small team.

Key Responsibilities

• Design and develop Spark applications using PySpark for enterprise-scale data processing.
• Build and maintain ETL data pipelines feeding a complex data lake implementation.
• Implement Delta Lake (delta.io) for enterprise-grade data storage and reliability.
• Perform optimisation and performance tuning of Spark applications.
• Develop Spark Streaming and Structured Streaming workloads.
• Build and set up CI/CD pipelines for PySpark deployments.
• Write and execute test cases for Spark applications, including performance tests.
• Size Spark clusters and manage resources across Spark Standalone, YARN, and Kubernetes cluster managers.
• Set up and operate highly available Spark clusters with operational monitoring.
• Work with Agile application development teams to implement data strategies, build dataflows, and define conceptual data models.
• Forecast environment requirements based on anticipated demand from multiple application development teams.
• Create short-term plans to deliver environments supporting sprint-based development.
• Provide technical guidance and manage a small team of technical specialists.

Primary Skills (Must Have)

• Experience designing and developing Spark applications using PySpark.
• Hands-on experience building and maintaining ETL data pipelines.
• Expertise in Python development.
• Proficiency in writing SQL scripts.
• Spark Streaming and Structured Streaming knowledge is mandatory.
• Experience with Delta Lake (delta.io) for enterprise-grade implementation.
• Optimisation and performance tuning of Spark applications.
• Experience with different cluster managers:
• Spark Standalone
• YARN
• Kubernetes
• Spark cluster sizing and resource management for a complex data lake implementation.
• Experience setting up and operating highly available Spark clusters with operational monitoring.
• Experience building and setting up CI/CD pipelines for PySpark.
• Experience writing and executing test cases for Spark applications, including performance tests.
• Knowledge of Big Data on Cloud, preferably GCP services such as Dataproc and GCS.
• Strong communication skills and the ability to plan and prioritise own time effectively.
• Ability to manage a small team as technical specialists.

Secondary Skills (Nice To Have)

• Java or Scala development experience.
• AWS or Azure cloud platform experience.
• Exposure to workflow orchestration tools such as Airflow or Cloud Composer.
• Familiarity with data governance, lineage, and cataloguing practices.
• Observability tooling for Spark workloads – metrics, logging, and alerting.

GCP Data Engineer certification is an advantage

Requirements: