Senior PySpark Data Engineer – Big Data (BFSI)

V2 Solutions; Hinoba-an; On-site; PHP 796,000 - 1,194,000; Full time

Job summary

V2 Solutions is seeking a Big Data Technical Senior Associate Engineer to design and deliver scalable data products on Cloudera platforms. You will partner with Business, Risk, and Technology teams in the BFSI domain, building robust pipelines and ensuring data quality in batch and streaming modes.

The role emphasizes hands-on expertise with Hadoop/Spark, SQL, and cloud/enterprise data platforms, with on-site Mumbai as the location. Prior experience in onsite/offshore models is valued.

Qualifications

• Must have hands-on experience building data pipelines on Hadoop/Spark ecosystem.
• Strong Spark (Scala and/or PySpark), Hive/Impala SQL, and performance tuning.
• Kafka for streaming ingestion and NiFi/StreamSets for batch/near-real-time flows.
• Experience with Cloudera Manager, YARN/Tez, HDFS, and Oozie/Airflow.
• Data warehousing concepts: dimensional modelling, partitioning, bucketing.
• Linux/Unix, Shell scripting, Git, and CI/CD (Jenkins/GitLab CI).
• Strong SQL and data modelling for BFSI use cases (lending, liabilities, risk, reporting).
• Experience in writing HLD/LLD and unit/integration testing.
• Exposure to SDLC/Agile and onsite/offshore model.

Responsibilities

• Develop robust Spark jobs (batch and streaming) with unit tests and observability.
• Implement ingestion patterns (Kafka/NiFi), data quality checks, and job scheduling.
• Analyze and tune SQL/Spark for large-scale datasets.

Skills

Hadoop

Spark

Hive/Impala SQL

Kafka

NiFi/StreamSets

Cloudera Manager/YARN/Tez

HDFS

Oozie/Airflow

Data warehousing modeling

Linux/Unix

Shell scripting

Git

CI/CD (Jenkins/GitLab CI)

SQL

BFSI data modelling

HLD/LLD docs

Agile

Education

B.E./B.Tech or equivalent

Cloudera Certified Associate/Professional or CDP certifications

AWS/Azure/GCP data certifications (nice to have)

Databricks Lakehouse Fundamentals or Associate (nice to have)

Tools

Cloudera Data Platform

Docker

Kubernetes

Job description

V2 Solutions is seeking a Big Data Technical Senior Associate Engineer to design and deliver scalable data products on Cloudera platforms. You will partner with Business, Risk, and Technology teams in the BFSI domain, building robust pipelines and ensuring data quality in batch and streaming modes.

The role emphasizes hands-on expertise with Hadoop/Spark, SQL, and cloud/enterprise data platforms, with on-site Mumbai as the location. Prior experience in onsite/offshore models is valued.