Pyspark Developer ( Pune /bangalore/Kolkata)

Price Waterhouse Cooper LLP; Kolkata District, Pune District, Bengaluru; Hybrid; INR 1,200,000 - 1,800,000; Full time

Job summary

Price Waterhouse Cooper LLP is seeking a Big Data Technical Senior Associate Engineer with 4-6 years of experience to deliver scalable data products on Cloudera platforms. The role emphasizes hands-on data engineering, Spark projects, and BFSI domain knowledge in Mumbai.

The candidate will work with Business, Risk, and Technology stakeholders, implementing streaming ingestion patterns and ensuring data quality and observability in large-scale datasets.

Qualifications

• Hands-on experience building data pipelines on Hadoop/Spark ecosystem.
• Strong Spark (Scala and/or PySpark), Hive/Impala SQL, and performance tuning.
• Working knowledge of Kafka for streaming ingestion and NiFi (or StreamSets) for batch/near-real-time flows.
• Experience with Cloudera Manager, YARN/Tez, HDFS, and job orchestration using Oozie/Airflow.
• Good understanding of data warehousing concepts (dimensional modelling, partitioning, bucketing).
• Proficiency in Linux/Unix, Shell scripting, Git, and CI/CD (Jenkins/GitLab CI).
• Strong SQL and data modelling for BFSI use cases (lending, liabilities, risk, regulatory reporting).
• Experience in writing technical design documents (HLD/LLD) and unit/integration testing.
• Exposure to SDLC/Agile and working in onsiteoffshore model.
• Location - Mumbai.
• Develop robust Spark jobs (batch and streaming) with unit tests and observability.
• Implement ingestion patterns (Kafka/NiFi), data quality checks, and job scheduling.
• Analyze and tune SQL/Spark for large-scale datasets.

Responsibilities

• Develop robust Spark jobs (batch and streaming) with unit tests and observability.
• Implement ingestion patterns (Kafka/NiFi) and data quality checks.
• Tune SQL/Spark for large-scale datasets and optimize performance.
• Collaborate with Business, Risk, and Technology stakeholders to deliver scalable data products.

Skills

Spark

Hive/Impala SQL

Kafka

NiFi/StreamSets

Cloudera Manager

YARN/Tez

HDFS

Oozie/Airflow

SQL/Data modelling

Linux/Unix

Shell scripting

Git

CI/CD

Agile

Education

B.E./B.Tech or equivalent

Tools

Oozie

Airflow

NiFi

StreamSets

Job description

Big Data Technical Senior Associate Engineer

We are looking for a suitable candidate for the opening of the Senior/Associate Engineer role with good to have experience in the Banking and Financial Services domain with 4-6 years of relevant experience in Data Engineering / Big Data platforms. The candidate will work closely with Business, Risk, and Technology stakeholders to deliver scalable data products on Cloudera (CDH/CDP).

Key Roles & Responsibilities:

Must have / Primary Skills / Mandatory

• Hands-on experience building data pipelines on Hadoop/Spark ecosystem
• Strong Spark (Scala and/or PySpark), Hive/Impala SQL, and performance tuning
• Working knowledge of Kafka for streaming ingestion and NiFi (or StreamSets) for batch/near-real-time flows
• Experience with Cloudera Manager, YARN/Tez, HDFS, and job orchestration using Oozie/Airflow
• Good understanding of data warehousing concepts (dimensional modelling, partitioning, bucketing)
• Proficiency in Linux/Unix, Shell scripting, Git, and CI/CD (Jenkins/GitLab CI)
• Strong SQL and data modelling for BFSI use cases (lending, liabilities, risk, regulatory reporting)
• Experience in writing technical design documents (HLD/LLD) and unit/integration testing
• Exposure to SDLC/Agile and working in onsiteoffshore model
• Location - Mumbai
• Develop robust Spark jobs (batch and streaming) with unit tests and observability
• Implement ingestion patterns (Kafka/NiFi), data quality checks, and job scheduling
• Analyze and tune SQL/Spark for large-scale datasets

Good to have / Secondary Skills / Desired

• Experience with Cloudera Data Platform (CDP) Private Cloud Base/Public Cloud
• Security and governance: Apache Ranger, Atlas; Kerberos; Sentry (legacy)
• Cloud data services – AWS (EMR, Glue, S3), Azure (HDInsight, Synapse, ADLS), or GCP (Dataproc, BigQuery)
• Databricks experience (Spark, Delta Lake) for select workloads
• Containerization and orchestration (Docker/Kubernetes) for micro-batch/ML workloads
• Python for data processing and utilities; familiarity with Scala build tools (sbt/maven)
• Monitoring/observability – Cloudera Manager metrics, Grafana/Prometheus, log aggregation
• Experience with BI consumption patterns and semantic layers for risk/regulatory dashboards

Educational Qualifications

B.E./B.Tech or equivalent

Experience Range

4-6 years

Certifications (Preferred)

• Cloudera Certified Associate/Professional (CCA/CCP) or CDP certifications
• AWS/Azure/GCP data certifications (nice to have)
• Databricks Lakehouse Fundamentals or Associate (nice to have)