Lead Pyspark Developer

Luxoft India; Bengaluru; On-site; INR 3,800,000 - 7,000,000; Full time

Job summary

Luxoft India is seeking a Senior Data Engineer to join the Surveillance Enhancement Program. You will develop and maintain data-intensive PySpark applications, lead a development team, and design data pipelines to meet regulatory obligations. Strong Python, Pandas/Numpy, and RESTful API skills are required.

The role focuses on data sourcing, UI/workflow enhancements, and building robust data processing capabilities within a regulated financial domain.

Qualifications

• 10+ years of experience in data-intensive PySpark development.
• Experience leading a Development Team.
• Proven experience as a Core Python developer.
• Strong knowledge of Pandas/Numpy dataframes and data wrangling.
• Experience with RESTful services and APIs; Git version control.
• Familiar with concurrency and error handling in data pipelines.

Responsibilities

• Develop software applications based on business requirements.
• Maintain software applications and enhancements per project specifications.
• Participate in requirement analysis, design, development, testing, and implementation activities.
• Propose new techniques and technologies for software development.
• Perform unit testing and user acceptance testing to evaluate application functionality.
• Ensure completion of development tasks within deadlines; follow coding standards.
• Provide assistance to Junior Developers when needed; perform code reviews and recommend improvements.
• Review business requirements and translate into reliable applications.
• Develop coding documentation and technical specifications for assigned projects.
• Act as primary contact for development queries and concerns.
• Analyze and resolve development issues accurately.

Skills

Data-intensive PySpark development

Team leadership

Core Python development

Gen AI understanding

DB connectivity and data loading

Pandas/Numpy dataframes

Lambda Functions & Decorators

Dataframe/Series vector operations

applymap/apply/map functions

Concurrency and error handling in data

RESTful services and APIs

Git source control

Unit testing familiarity

Web backend: Django, PostgreSQL

Jira

Financial markets knowledge

MAR/MiFID/Dodd-Frank regulations

Tools

Git

Jira

Django

PostgreSQL

Job description

Surveillance Enhancement Program (SEP) is a multi-year program (MYP) to build and enhance the surveillance platform to detect potential instances of market misconduct, market manipulation and market abuse. The Bank met its commitment to FCA (Financial Conduct Authority, FCA) to meet Market Abuse Regulation ("MAR") requirements for critical data sources in September 2017. Following that, subsequent projects were initiated to further enhance, expand and complement the current coverage of the automated surveillance platform. This project will focus on Alerts enhancements, Strategic data sourcing from FM order and trade lakes, UI / workflow enhancements to meet regulatory obligations (i.e. Market Abuse Regulation, Dodd Frank). This also includes Control Room & Suitability Monitoring related data, UI and scenario enhancements.

Develop and maintain software applications for the Surveillance Enhancement Program (SEP), participate in the full software development lifecycle, and lead enhancements to alerts, data sourcing, and UI/workflow to meet regulatory obligations.

Responsibilities

• Develop software applications based on business requirements
• Maintain software applications and make enhancements according to project specifications
• Participate in requirement analysis, design, development, testing, and implementation activities
• Propose new techniques and technologies for software development.
• Perform unit testing and user acceptance testing to evaluate application functionality
• Ensure to complete the assigned development tasks within the deadlines
• Work in compliance with coding standards and best practices
• Provide assistance to Junior Developers when needed.
• Perform code reviews and recommend improvements.
• Review business requirements and recommend changes to develop reliable applications.
• Develop coding documentation and other technical specifications for assigned projects.
• Act as primary contact for development queries and concerns.
• Analyze and resolve development issues accurately.

Required Skills

• 10+ years of experience in data intensive Pyspark development
• Must have experience in leading a Development Team
• Proven experience as a Core Python developer
• Good understanding of Gen AI
• Strong knowledge of DB connectivity, data loading , transformation, calculation
• Extensive experience in Pandas/Numpy dataframes, slicing, data wrangling, aggregations
• Lambda Functions, Decorators
• Vector operations on Pandas dataframes /series
• Application of applymap, apply, map functions
• Concurrency and error handling data pipeline batch of size [1-10 gb]
• Ability to understand business requirements and translate them into technical requirements
• Ability to design architecture of data pipeline for concurrent data processing
• Familiar with creating/designing RESTful services and APIs
• Familiar with application unit tests
• Working with Git source control Service-orientated architecture, including the ability to consider integrations with other applications and services
• Debugging application

Preferred Skills

• Knowledge of web backend technology - Django, Python, PostgreSQL.
• Atlassian Jira
• Understanding of Financial Markets Asset Classes (FX, FI, Equities, Rates, Commodities & Credit), various trade types (OTC, exchange traded, Spot, Forward, Swap, Options) and related systems is a plus
• Surveillance domain knowledge, regulations (MAR, MIFID, CAT, Dodd Frank) and related Systems knowledge is certainly a plus