Senior PySpark Developer

Basebiz; Chennai District, Bengaluru; On-site; INR 2,400,000 - 3,600,000; Full time

Job summary

Basebiz in India seeks a Senior PySpark Developer to lead data-intensive PySpark development and core Python work for a Trade Surveillance team. The role requires strong proficiency in data processing, pipelines, concurrency, RESTful services, testing, and debugging across Bengaluru and Chennai, with 8+ years of experience.

Hands-on experience with Pandas/NumPy, Lambda/Decorators, and SOA is essential, along with knowledge of MAR-style regulatory considerations.

Qualifications

• 8+ years of experience in data-intensive PySpark development.
• Strong experience as a Core Python Developer.
• Basic experience in Generative AI / Agentic AI.
• Strong understanding of Classes, OOPS, exception handling, and parallel processing.
• Strong knowledge of database connectivity, data loading, data transformation, and calculations.
• Extensive experience with Pandas/NumPy DataFrames, including slicing, data wrangling, and aggregations.
• Hands-on experience with Lambda Functions and Decorators.
• Experience with vector operations on Pandas DataFrames/Series.
• Experience using applymap, apply, and map functions.
• Concurrency and error handling for data pipeline batches.
• Ability to translate business requirements into technical requirements.
• Design architecture of data pipelines for concurrent processing.
• Familiarity with RESTful services and APIs.
• Familiarity with application unit testing.
• Experience with Git source control.
• Understanding of service‑oriented architecture.
• Strong application debugging skills.

Responsibilities

• Develop software applications based on business requirements.
• Maintain software applications and make enhancements according to project specifications.
• Participate in requirement analysis, design, development, testing, and implementation activities.
• Propose new techniques and technologies for software development.
• Perform unit testing and user acceptance testing to evaluate application functionality.
• Ensure assigned development tasks are completed within the defined deadlines.
• Work in compliance with coding standards and best practices.
• Provide assistance to Junior Developers when required.
• Perform code reviews and recommend improvements.
• Review business requirements and recommend changes to develop reliable applications.
• Develop coding documentation and technical specifications for assigned projects.
• Act as the primary contact for development queries and concerns.
• Analyze and resolve development issues accurately.

Skills

data-intensive PySpark development

Core Python

Generative AI

OOPS and parallel processing

database connectivity

Pandas/NumPy DataFrames

Lambda/Decorators

vector operations on DataFrames

applymap/apply/map

concurrency and error handling

translate requirements to technical

data pipeline architecture

RESTful APIs

unit testing

Git

SOA concepts

debugging

Tools

Apache Airflow

Jira

Job description

123891

Senior PySpark Developer

Locations: Bengaluru and Chennai, India
Experience: 8+ Years
Domain: Trade Surveillance Investment Banking

Project Description

We are looking for a Senior PySpark Developer to work with a leading investment bank client in the Trade Surveillance Team. The role requires strong experience in data-intensive PySpark development and core Python development, along with hands‑on expertise in data processing, data pipelines, concurrency, RESTful services, application development, testing, and debugging.

Skills Required

• 8+ years of experience in data-intensive PySpark development.
• Strong experience as a Core Python Developer.
• Basic experience in Generative AI / Agentic AI.
• Strong understanding of Classes, OOPS, exception handling, and parallel processing.
• Strong knowledge of database connectivity, data loading, data transformation, and calculations.
• Extensive experience with Pandas/NumPy DataFrames, including slicing, data wrangling, and aggregations.
• Hands‑on experience with Lambda Functions and Decorators.
• Experience with vector operations on Pandas DataFrames/Series.
• Experience using applymap, apply, and map functions.
• Experience with concurrency and error handling for data pipeline batches ranging from 110 GB.
• Ability to understand business requirements and translate them into technical requirements.
• Ability to design the architecture of data pipelines for concurrent data processing.
• Familiarity with creating/designing RESTful services and APIs.
• Familiarity with application unit testing.
• Experience working with Git source control.
• Understanding of service‑oriented architecture, including the ability to consider integrations with other applications and services.
• Strong application debugging skills.

Nice to Have

• Knowledge of web backend technologies such as Django, Python, and PostgreSQL.
• Experience with Apache Airflow.
• Experience with Atlassian Jira.
• Understanding of Financial Markets Asset Classes, including:
  • FX
  • FI
  • Equities
  • Rates
  • Commodities
  • Credit
• Understanding of various trade types, including:
  • OTC
  • Exchange Traded
  • Spot
  • Forward
  • Swap
  • Options
• Knowledge of Trade Surveillance domain, regulations such as MAR, MIFID, CAT, and Dodd Frank, and related systems is a plus.

Responsibilities

• Develop software applications based on business requirements.
• Maintain software applications and make enhancements according to project specifications.
• Participate in requirement analysis, design, development, testing, and implementation activities.
• Propose new techniques and technologies for software development.
• Perform unit testing and user acceptance testing to evaluate application functionality.
• Ensure assigned development tasks are completed within the defined deadlines.
• Work in compliance with coding standards and best practices.
• Provide assistance to Junior Developers when required.
• Perform code reviews and recommend improvements.
• Review business requirements and recommend changes to develop reliable applications.
• Develop coding documentation and technical specifications for assigned projects.
• Act as the primary contact for development queries and concerns.
• Analyze and resolve development issues accurately.