Senior Software Engineer - Data Pipelines Lead (Pyspark)

Luxoft; Gdańsk; Vor Ort; PLN 300.000 - 420.000; Vollzeit

Benefits dieser Stelle

Laptop and monitor equipment

Zusammenfassung

Luxoft in Gdańsk is seeking an experienced Senior Data/Software Engineer for the Surveillance Enhancement Program. You will develop and enhance a regulatory surveillance platform, focusing on data intensive PySpark workloads and UI/workflow improvements to meet MAR/Dodd-Frank obligations.

Responsibilities include design, development, testing, and optimization, mentoring juniors, and ensuring delivery within deadlines. Strong Python, PySpark, and data engineering skills are required.

Qualifikationen

• 10+ years of experience in data intensive PySpark development
• Experience leading a development team
• Proven experience as a Core Python developer
• Good understanding of Gen AI
• Strong knowledge of DB connectivity, data loading, transformation, calculation
• Extensive experience in Pandas/Numpy dataframes, slicing, data wrangling, aggregations
• Vector operations on Pandas dataframes/series
• Apply/map functions usage
• Concurrency and error handling in data pipelines (1-10 GB batches)
• Ability to translate business requirements into technical designs
• Design data pipeline architectures for concurrent processing
• Experience with Git and SOA/integrations

Aufgaben

• Develop software applications based on business requirements
• Maintain and enhance applications per project specs
• Participate in requirements analysis, design, development, testing, and implementation
• Propose new techniques and technologies for software development
• Perform unit testing and UAT to verify functionality
• Complete development tasks within deadlines
• Adhere to coding standards and best practices
• Mentor junior developers as needed
• Review code and suggest improvements
• Refine requirements into reliable tech solutions
• Create coding documentation and technical specs
• Act as primary contact for development queries
• Analyze and resolve development issues accurately

Kenntnisse

Data-intensive PySpark

Window Functions

Performance Optimization

Spark Architecture

Salting

Column Pruning

Leading a Development Team

Core Python

Gen AI

DB Connectivity

Pandas/Numpy

Vector operations

Apply/Map Functions

Concurrency & Error Handling

Data Pipeline Architecture

Git

SOA/Integrations

Tools

Python

PySpark

Jobbeschreibung

Luxoft in Gdańsk is seeking an experienced Senior Data/Software Engineer for the Surveillance Enhancement Program. You will develop and enhance a regulatory surveillance platform, focusing on data intensive PySpark workloads and UI/workflow improvements to meet MAR/Dodd-Frank obligations.

Responsibilities include design, development, testing, and optimization, mentoring juniors, and ensuring delivery within deadlines. Strong Python, PySpark, and data engineering skills are required.