Lead PySpark Data Engineer - Regulatory Surveillance

Luxoft; Województwo mazowieckie; On-site; PLN 180,000 - 260,000; Full time

Benefits offered by this job

Laptop

Monitor

Job summary

Luxoft in Poland is seeking an experienced Senior PySpark Developer to drive data-intensive solutions. You will lead a development team, design scalable data pipelines, and implement performance optimizations using PySpark, Python, and Pandas/Numpy, aligning with regulatory and industry demands.

You will collaborate across engineering to deliver robust, scalable software, participate in design reviews, and mentor junior developers while ensuring best practices and timely delivery.

Qualifications

• 10+ years of data intensive PySpark development and related techniques.
• Experience leading a development team.
• Proven core Python development experience.
• Strong understanding of Gen AI and data pipelines.
• Deep knowledge of DB connectivity, data loading and transformation.
• Extensive Pandas/Numpy dataframe operations and data wrangling.

Responsibilities

• Develop software applications based on business requirements.
• Maintain applications and implement enhancements per specs.
• Participate in requirement analysis, design, development, testing, and deployment.
• Propose new techniques and technologies for software development.
• Perform unit testing and UAT to verify functionality.
• Complete development tasks within deadlines.
• Adhere to coding standards and best practices.
• Assist junior developers when needed.
• Conduct code reviews and suggest improvements.
• Review requirements and translate to technical solutions.
• Create coding documentation and technical specs.
• Act as primary contact for development queries.
• Analyze and resolve development issues.

Skills

PySpark

Window Functions

Performance Optimization

Spark Architecture

Salting

Column Pruning

Lead Development Team

Core Python

Gen AI

DB Connectivity

Data Loading

Data Transformation

Calculation

Pandas/Numpy

Slicing

Data Wrangling

Aggregations

Vector Operations

Apply/Map Functions

Concurrency

Error Handling

Git

Tools

Git

Job description

Luxoft in Poland is seeking an experienced Senior PySpark Developer to drive data-intensive solutions. You will lead a development team, design scalable data pipelines, and implement performance optimizations using PySpark, Python, and Pandas/Numpy, aligning with regulatory and industry demands.

You will collaborate across engineering to deliver robust, scalable software, participate in design reviews, and mentor junior developers while ensuring best practices and timely delivery.