Lead Pyspark Developer

Luxoft; Gdańsk; On-site; PLN 180,000 - 300,000; Full time

Benefits offered by this job

Private Medical care

Dental care

Life Insurance

Internal Mobility program

Job summary

Luxoft in Poland is seeking a senior data engineering professional to lead the development of the Surveillance Enhancement Program (SEP). You will design and optimize PySpark-based data pipelines, integrate Gen AI capabilities, and collaborate with cross-functional teams across projects and locations.

The role requires 10+ years in data-intensive PySpark development, strong Python skills, and experience mentoring developers.

Qualifications

• 10+ years of experience in data-intensive PySpark development and Spark architecture.
• Experience leading a development team.
• Proven experience as a core Python developer.
• Strong knowledge of DB connectivity, data loading, transformation, calculation.
• Extensive experience with Pandas and NumPy dataframes.
• Knowledge of Gen AI.
• Concurrency and error handling in data pipelines (1-10 GB batches).
• Ability to understand business requirements and translate them into technical requirements.
• Experience with Git and service-oriented architecture and integrations.

Responsibilities

• Develop software applications based on business requirements.
• Maintain software applications and make enhancements according to project specifications.
• Participate in requirement analysis, design, development, testing, and implementation activities.
• Propose new techniques and technologies for software development.
• Perform unit testing and user acceptance testing to evaluate application functionality.
• Ensure to complete the assigned development tasks within the deadlines.
• Work in compliance with coding standards and best practices.
• Provide assistance to Junior Developers when needed.
• Perform code reviews and recommend improvements.
• Review business requirements and recommend changes to develop reliable applications.
• Develop coding documentation and other technical specifications for assigned projects.
• Act as primary contact for development queries and concerns.
• Analyze and resolve development issues accurately.

Skills

PySpark development

Spark architecture

Window functions

Performance optimization

Salting

Column pruning

Lead a development team

Core Python

Gen AI

DB connectivity

Pandas

NumPy

Dataframes

Vector operations

Concurrency

Apply/map functions

Git

SOA architecture

Integrations

Requirements translation

Tools

Django

PostgreSQL

Job description

Private Medical & Dental care & Life Insurance covered

Internal Mobility program - possibility of rotation between projects, locations, accounts

...and more!

Project Description

The Surveillance Enhancement Program (SEP) is a multi-year initiative aimed at developing and improving the surveillance platform to identify potential market misconduct, manipulation, and abuse. Our client fulfilled its commitment to the Financial Conduct Authority (FCA) by meeting Market Abuse Regulation (MAR) requirements for critical data sources in September 2017.

Subsequent projects have been launched to further enhance, expand, and complement the existing automated surveillance platform. This project will focus on alert enhancements, strategic data sourcing from FM order and trade lakes, and user interface and workflow improvements to comply with regulatory obligations such as Market Abuse Regulation and Dodd-Frank.

Additionally, the scope includes enhancements related to Control Room and Suitability Monitoring data, interfaces, and scenarios.

Responsibilities

• Develop software applications based on business requirements
• Maintain software applications and make enhancements according to project specifications
• Participate in requirement analysis, design, development, testing, and implementation activities
• Propose new techniques and technologies for software development.
• Perform unit testing and user acceptance testing to evaluate application functionality
• Ensure to complete the assigned development tasks within the deadlines
• Work in compliance with coding standards and best practices
• Provide assistance to Junior Developers when needed.
• Perform code reviews and recommend improvements.
• Review business requirements and recommend changes to develop reliable applications.
• Develop coding documentation and other technical specifications for assigned projects.
• Act as primary contact for development queries and concerns.
• Analyze and resolve development issues accurately.

Mandatory Skills Description

• 10+ years of experience in data intensive Pyspark development ,Window Functions , Performance Optimization ,Spark Architecture,Salting ,Column Pruning
• Must have experience in leading a Development Team
• Proven experience as a Core Python developer
• Good understanding of Gen AI
• Strong knowledge of DB connectivity, data loading , transformation, calculation
• Extensive experience in Pandas/Numpy dataframes, slicing, data wrangling, aggregations
• Vector operations on Pandas dataframes /series
• Application of apply map, apply, map functions
• Concurrency and error handling data pipeline batch of size [1-10 gb]
• Ability to understand business requirements and translate them into technical requirements
• Ability to design architecture of data pipeline for concurrent data processing
• Working with Git source control Service-orientated architecture, including the ability to consider integrations with other applications and services

Nice-to-Have Skills Description

• Knowledge of web backend technology - Django, Python, PostgreSQL.