Lead Pyspark Developer

Luxoft; Warszawa; On-site; PLN 400,000 - 640,000; Full time

Job summary

Luxoft in Warsaw is seeking a senior data engineering lead to develop and optimize data-intensive PySpark applications. You will guide a development team, design scalable data pipelines, and ensure high-quality code and documentation.

The role requires strong Python expertise, experience with Pandas/Numpy, and knowledge of Spark architecture and performance tuning. Collaboration with cross-functional teams and adherence to standards are essential.

Qualifications

• 10+ years of experience in data intensive PySpark development with Spark architecture and optimization techniques.
• Experience leading a development team.
• Proven Python developer with strong Pandas/Numpy dataframes proficiency.
• Good understanding of Gen AI and data-driven applications.
• Strong DB connectivity, data loading, transformation and calculation skills.
• Experience with Pandas/Numpy dataframes, slicing, wrangling and aggregations.
• Familiarity with vector operations on dataframes/series and apply/map usage.
• Experience with concurrency and error handling in data pipelines (1–10 GB batches).
• Ability to translate business requirements into technical specs and design scalable data pipelines.
• Knowledge of Git and service-oriented architecture with potential integrations.

Responsibilities

• Develop software applications based on business requirements.
• Maintain software applications and implement enhancements per project specs.
• Participate in requirement analysis, design, development, testing, and deployment activities.
• Propose new techniques and technologies for software development.
• Perform unit testing and user acceptance testing to verify functionality.
• Complete assigned development tasks within deadlines.
• Adhere to coding standards and best practices.
• Provide guidance to junior developers when needed.
• Perform code reviews and recommend improvements.
• Review requirements and suggest changes to build reliable applications.
• Develop coding documentation and other technical specifications for projects.
• Act as a primary contact for development queries and concerns.
• Analyze and resolve development issues accurately.

Skills

PySpark data engineering

Spark architecture

Window functions

Data wrangling (Pandas/Numpy)

DB connectivity

Performance optimization

Salting

Column pruning

Team lead experience

Gen AI understanding

Data pipelines concurrency

Git version control

Python backend (Django)

PostgreSQL

Job description

DXC Luxoft is a global IT services and consulting company, part of DXC Technology, delivering cutting-edge digital and software solutions worldwide. We partner with leading financial institutions and global enterprises, helping them transform and scale through modern technology.

Join a global community of engineers and be part of building impactful, modern technology solutions!

Project Description:

The Surveillance Enhancement Program (SEP) is a multi-year initiative aimed at developing and improving the surveillance platform to identify potential market misconduct, manipulation, and abuse. Our client fulfilled its commitment to the Financial Conduct Authority (FCA) by meeting Market Abuse Regulation (MAR) requirements for critical data sources in September 2017.

Subsequent projects have been launched to further enhance, expand, and complement the existing automated surveillance platform. This project will focus on alert enhancements, strategic data sourcing from FM order and trade lakes, and user interface and workflow improvements to comply with regulatory obligations such as Market Abuse Regulation and Dodd-Frank.

Additionally, the scope includes enhancements related to Control Room and Suitability Monitoring data, interfaces, and scenarios.

Responsibilities:

• Develop software applications based on business requirements
• Maintain software applications and make enhancements according to project specifications
• Participate in requirement analysis, design, development, testing, and implementation activities
• Propose new techniques and technologies for software development.
• Perform unit testing and user acceptance testing to evaluate application functionality
• Ensure to complete the assigned development tasks within the deadlines
• Work in compliance with coding standards and best practices
• Provide assistance to Junior Developers when needed.
• Perform code reviews and recommend improvements.
• Review business requirements and recommend changes to develop reliable applications.
• Develop coding documentation and other technical specifications for assigned projects.
• Act as primary contact for development queries and concerns.
• Analyze and resolve development issues accurately.

Mandatory Skills Description:

• 10+ years of experience in data intensive Pyspark development ,Window Functions , Performance Optimization ,Spark Architecture,Salting ,Column Pruning
• Must have experience in leading a Development Team
• Proven experience as a Core Python developer
• Good understanding of Gen AI
• Strong knowledge of DB connectivity, data loading , transformation, calculation
• Extensive experience in Pandas/Numpy dataframes, slicing, data wrangling, aggregations
• Vector operations on Pandas dataframes /series
• Application of apply map, apply, map functions
• Concurrency and error handling data pipeline batch of size [1-10 gb]
• Ability to understand business requirements and translate them into technical requirements
• Ability to design architecture of data pipeline for concurrent data processing
• Working with Git source control Service-orientated architecture, including the ability to consider integrations with other applications and services

Nice-to-Have Skills Description:

• Knowledge of web backend technology - Django, Python, PostgreSQL.