Pyspark Developer

Price Waterhouse Cooper LLP; Kolkata District, Pune District, Bengaluru; Hybrid; INR 1,500,000 - 2,300,000; Full time

Job summary

Price Waterhouse Cooper LLP invites a Big Data Technical Senior Associate Engineer with 4–6 years of experience in Data Engineering to build scalable data products on Hadoop/Spark for the BFSI sector. You will collaborate with Business, Risk, and Tech teams to deliver robust data pipelines and ensure quality across batch and streaming workloads.

The role emphasizes hands-on development, design documentation, and adherence to SDLC/Agile practices, with on-site/offshore collaboration and a

Qualifications

• Hands-on data pipelines in Hadoop/Spark stack.
• Strong Spark with Scala and/or PySpark; Hive/Impala SQL.
• Kafka for streaming; NiFi/StreamSets for batch/near-real-time.
• Cloudera Manager, YARN/Tez, HDFS; Oozie/Airflow orchestration.
• Data warehousing concepts with dimensional modelling.
• Linux/Unix, Shell scripting, Git, CI/CD (Jenkins/GitLab CI).
• Strong SQL and data modelling for BFSI use cases.
• Experience writing HLD/LLD and unit/integration tests.
• Exposure to SDLC/Agile; onsite/offshore model.
• Location - Mumbai

Skills

Spark (Scala/PySpark)

Hive/Impala SQL

Kafka streaming

NiFi/StreamSets

Cloudera Manager

YARN/Tez

HDFS

Oozie/Airflow

Dimensional modelling

Linux/Unix

CI/CD (Jenkins/GitLab CI)

SQL data modelling BFSI

HLD/LLD & testing

SDLC/Agile onsite/offshore

Education

B.E./B.Tech or equivalent

Tools

Cloudera Manager

Oozie

Airflow

YARN/Tez

HDFS

Kafka

Job description

Big Data Technical Senior Associate Engineer

We are looking for a suitable candidate for the opening of the Senior/Associate Engineer role with good to have experience in the Banking and Financial Services domain with 4-6 years of relevant experience in Data Engineering / Big Data platforms. The candidate will work closely with Business, Risk, and Technology stakeholders to deliver scalable data products on Cloudera (CDH/CDP).

Key Roles & Responsibilities:

Must have / Primary Skills / Mandatory

• Hands-on experience building data pipelines on Hadoop/Spark ecosystem
• Strong Spark (Scala and/or PySpark), Hive/Impala SQL, and performance tuning
• Working knowledge of Kafka for streaming ingestion and NiFi (or StreamSets) for batch/near-real-time flows
• Experience with Cloudera Manager, YARN/Tez, HDFS, and job orchestration using Oozie/Airflow
• Good understanding of data warehousing concepts (dimensional modelling, partitioning, bucketing)
• Proficiency in Linux/Unix, Shell scripting, Git, and CI/CD (Jenkins/GitLab CI)
• Strong SQL and data modelling for BFSI use cases (lending, liabilities, risk, regulatory reporting)
• Experience in writing technical design documents (HLD/LLD) and unit/integration testing
• Exposure to SDLC/Agile and working in onsiteoffshore model
• Location - Mumbai
• Develop robust Spark jobs (batch and streaming) with unit tests and observability
• Implement ingestion patterns (Kafka/NiFi), data quality checks, and job scheduling
• Analyze and tune SQL/Spark for large-scale datasets

Good to have / Secondary Skills / Desired

• Experience with Cloudera Data Platform (CDP) Private Cloud Base/Public Cloud
• Security and governance: Apache Ranger, Atlas; Kerberos; Sentry (legacy)
• Cloud data services - AWS (EMR, Glue, S3), Azure (HDInsight, Synapse, ADLS), or GCP (Dataproc, BigQuery)
• Databricks experience (Spark, Delta Lake) for select workloads
• Containerization and orchestration (Docker/Kubernetes) for micro-batch/ML workloads
• Python for data processing and utilities; familiarity with Scala build tools (sbt/maven)
• Monitoring/observability - Cloudera Manager metrics, Grafana/Prometheus, log aggregation
• Experience with BI consumption patterns and semantic layers for risk/regulatory dashboards

Educational Qualifications

B.E./B.Tech or equivalent

Experience Range

4-6 years

Certifications (Preferred)

• Cloudera Certified Associate/Professional (CCA/CCP) or CDP certifications
• AWS/Azure/GCP data certifications (nice to have)
• Databricks Lakehouse Fundamentals or Associate (nice to have)