Pyspark Developer ( Mumbai)

TymblHub; Hinoba-an; Vor Ort; PHP 796.000 - 1.194.000; Vollzeit

Zusammenfassung

V2 Solutions is seeking a Big Data Technical Senior Associate Engineer to design and deliver scalable data products on Cloudera platforms. You will partner with Business, Risk, and Technology teams in the BFSI domain, building robust pipelines and ensuring data quality in batch and streaming modes.

The role emphasizes hands-on expertise with Hadoop/Spark, SQL, and cloud/enterprise data platforms, with on-site Mumbai as the location. Prior experience in onsite/offshore models is valued.

Qualifikationen

• Must have hands-on experience building data pipelines on Hadoop/Spark ecosystem.
• Strong Spark (Scala and/or PySpark), Hive/Impala SQL, and performance tuning.
• Kafka for streaming ingestion and NiFi/StreamSets for batch/near-real-time flows.
• Experience with Cloudera Manager, YARN/Tez, HDFS, and Oozie/Airflow.
• Data warehousing concepts: dimensional modelling, partitioning, bucketing.
• Linux/Unix, Shell scripting, Git, and CI/CD (Jenkins/GitLab CI).
• Strong SQL and data modelling for BFSI use cases (lending, liabilities, risk, reporting).
• Experience in writing HLD/LLD and unit/integration testing.
• Exposure to SDLC/Agile and onsite/offshore model.

Aufgaben

• Develop robust Spark jobs (batch and streaming) with unit tests and observability.
• Implement ingestion patterns (Kafka/NiFi), data quality checks, and job scheduling.
• Analyze and tune SQL/Spark for large-scale datasets.

Kenntnisse

Hadoop

Spark

Hive/Impala SQL

Kafka

NiFi/StreamSets

Cloudera Manager/YARN/Tez

HDFS

Oozie/Airflow

Data warehousing modeling

Linux/Unix

Shell scripting

Git

CI/CD (Jenkins/GitLab CI)

SQL

BFSI data modelling

HLD/LLD docs

Agile

Ausbildung

B.E./B.Tech or equivalent

Cloudera Certified Associate/Professional or CDP certifications

AWS/Azure/GCP data certifications (nice to have)

Databricks Lakehouse Fundamentals or Associate (nice to have)

Tools

Cloudera Data Platform

Docker

Kubernetes

Jobbeschreibung

Big Data Technical Senior Associate Engineer

We are looking for a suitable candidate for the opening of the Senior/Associate Engineer role with good to have experience in the Banking and Financial Services domain with 4-6 years of relevant experience in Data Engineering / Big Data platforms. The candidate will work closely with Business, Risk, and Technology stakeholders to deliver scalable data products on Cloudera (CDH/CDP).

Key Roles & Responsibilities

• Must have / Primary Skills / Mandatory Hands-on experience building data pipelines on Hadoop/Spark ecosystem
• Strong Spark (Scala and/or PySpark), Hive/Impala SQL, and performance tuning
• Working knowledge of Kafka for streaming ingestion and NiFi (or StreamSets) for batch/near-real-time flows
• Experience with Cloudera Manager, YARN/Tez, HDFS, and job orchestration using Oozie/Airflow
• Good understanding of data warehousing concepts (dimensional modelling, partitioning, bucketing)
• Proficiency in Linux/Unix, Shell scripting, Git, and CI/CD (Jenkins/GitLab CI)
• Strong SQL and data modelling for BFSI use cases (lending, liabilities, risk, regulatory reporting)
• Experience in writing technical design documents (HLD/LLD) and unit/integration testing
• Exposure to SDLC/Agile and working in onsiteoffshore model

Location - Mumbai

• Develop robust Spark jobs (batch and streaming) with unit tests and observability
• Implement ingestion patterns (Kafka/NiFi), data quality checks, and job scheduling
• Analyze and tune SQL/Spark for large-scale datasets

Good to have / Secondary Skills / Desired Experience

• with Cloudera Data Platform (CDP)
• Private Cloud Base/Public Cloud Security and governance: Apache Ranger, Atlas; Kerberos; Sentry (legacy)
• Cloud data services – AWS (EMR, Glue, S3), Azure (HDInsight, Synapse, ADLS), or GCP (Dataproc, BigQuery)
• Databricks experience (Spark, Delta Lake) for select workloads
• Containerization and orchestration (Docker/Kubernetes) for micro-batch/ML workloads
• Python for data processing and utilities; familiarity with Scala build tools (sbt/maven)
• Monitoring/observability – Cloudera Manager metrics, Grafana/Prometheus, log aggregation
• Experience with BI consumption patterns and semantic layers for risk/regulatory dashboards

Educational Qualifications

• B.E./B.Tech or equivalent

Experience Range 4-6 years

Certifications (Preferred)

• Cloudera Certified Associate/Professional (CCA/CCP) or CDP certifications
• AWS/Azure/GCP data certifications (nice to have)
• Databricks Lakehouse Fundamentals or Associate (nice to have)