Lead Pyspark Developer

Luxoft; Województwo mazowieckie; Presencial; PLN 180 000 - 260 000; Tempo integral

Resumo da oferta

Luxoft in Poland is seeking a senior data engineer to lead the Surveillance Enhancement Program (SEP) focusing on PySpark data pipelines, MAR regulation compliance, and UI/workflow enhancements.

You will design architectures for large-scale data processing, optimize performance, and mentor a development team while collaborating with stakeholders. Knowledge of Python, Django, PostgreSQL, and Airflow is a plus.

Qualificações

• 15+ years of experience in data-intensive PySpark development and related tech.
• Experience leading a development team.
• Proven Python development expertise with OOP and exceptions.
• Strong DB connectivity, data loading and transformations.
• Solid Pandas/Numpy dataframes experience.
• Familiarity with Git-based workflows and SOA concepts.

Responsabilidades

• Develop software applications based on business requirements.
• Maintain and enhance applications per project specs.
• Lead requirement analysis, design, development, testing and implementation.
• Propose new techniques and technologies for software development.
• Perform unit and user acceptance testing.
• Ensure deadlines and coding standards are met.
• Provide guidance to junior developers and conduct code reviews.
• Review business requirements and translate to reliable apps.
• Develop coding documentation and technical specs.
• Serve as primary development query contact.
• Analyze and resolve development issues.

Conhecimentos

Pyspark development

Window functions

Performance optimization

Spark architecture

Salting

Column pruning

Lead development team

Core Python

Gen AI understanding

DB connectivity

Pandas/Numpy

Git

Django

PostgreSQL

Apache Airflow

Jira

Ferramentas

Git

Django

PostgreSQL

Apache Airflow

Jira

Descrição da oferta de emprego

Project description

Surveillance Enhancement Program (SEP) is a multi-year program (MYP) to build and enhance the surveillance platform to detect potential instances of market misconduct, market manipulation and market abuse. The Bank met its commitment to FCA (Financial Conduct Authority, FCA) to meet Market Abuse Regulation ("MAR") requirements for critical data sources in September 2017. Following that, subsequent projects were initiated to further enhance, expand and complement the current coverage of the automated surveillance platform. This project will focus on Alerts enhancements, Strategic data sourcing from FM order and trade lakes, UI / workflow enhancements to meet regulatory obligations (i.e. Market Abuse Regulation, Dodd Frank). This also includes Control Room & Suitability Monitoring related data, UI and scenario enhancements.

Responsibilities

• Develop software applications based on business requirements
• Maintain software applications and make enhancements according to project specifications
• Participate in requirement analysis, design, development, testing, and implementation activities
• Propose new techniques and technologies for software development.
• Perform unit testing and user acceptance testing to evaluate application functionality
• Ensure to complete the assigned development tasks within the deadlines
• Work in compliance with coding standards and best practices
• Provide assistance to Junior Developers when needed.
• Perform code reviews and recommend improvements.
• Review business requirements and recommend changes to develop reliable applications.
• Develop coding documentation and other technical specifications for assigned projects.
• Act as primary contact for development queries and concerns.
• Analyze and resolve development issues accurately.

SKILLS

Must have

• 15+ years of experience in data intensive Pyspark development ,Window Functions , Performance Optimization ,Spark Architecture,Salting ,Column Pruning
• Must have experience in leading a Development Team
• Proven experience as a Core Python developer
• Good understanding of Gen AI
• Experience developing Classes, OOPS, exception handling, parallel processing
• Strong knowledge of DB connectivity, data loading , transformation, calculation
• Extensive experience in Pandas/Numpy dataframes, slicing, data wrangling, aggregations
• Lambda Functions, Decorators
• Vector operations on Pandas dataframes /series
• Application of apply map, apply, map functions
• Concurrency and error handling data pipeline batch of size [1-10 gb]
• Ability to understand business requirements and translate them into technical requirements
• Ability to design architecture of data pipeline for concurrent data processing
• Working with Git source control Service-orientated architecture, including the ability to consider integrations with other applications and services
• Debugging application

Nice to have

• Knowledge of web backend technology - Django, Python, PostgreSQL.
• Apache Airflow
• Atlassian Jira
• Understanding of Financial Markets Asset Classes (FX, FI, Equities, Rates, Commodities & Credit), various trade types (OTC, exchange traded, Spot, Forward, Swap, Options) and related systems is a plus
• Surveillance domain knowledge, regulations (MAR, MIFID, CAT, Dodd Frank) and related Systems knowledge is certainly a plus