Senior PySpark Lead Developer - Data Pipeline Architect

Luxoft; Województwo mazowieckie; On-site; PLN 360,000 - 540,000; Full time

Job summary

Luxoft is seeking a senior data engineering professional to develop and optimize data-intensive PySpark applications for a surveillance platform.

You will lead a development team, design scalable data pipelines, and implement robust processing with focus on performance and reliability. Strong Python expertise and familiarity with Gen AI are essential to translate business requirements into technical solutions.

Qualifications

• 10+ years of data-intensive PySpark development experience.
• Proven experience leading a development team.
• Strong Python (core) programming skills.
• Good understanding of Gen AI and its application to data pipelines.
• Experience with Pandas/Numpy for data wrangling and analysis.
• Strong knowledge of DB connectivity, data loading, transformation, and calculations.

Responsibilities

• Develop software applications based on business requirements.
• Maintain and enhance applications per project specs.
• Participate in design, development, testing and implementation activities.
• Propose new techniques and technologies for software development.
• Perform unit testing and user acceptance testing of applications.
• Ensure task delivery within deadlines and coding standards.

Skills

PySpark

Data engineering

Window functions

Performance optimization

Spark architecture

Salting

Column pruning

Core Python

Gen AI

Pandas/Numpy

Git

Team leadership

Tools

Django

PostgreSQL

Apache Airflow

Job description

Luxoft is seeking a senior data engineering professional to develop and optimize data-intensive PySpark applications for a surveillance platform.

You will lead a development team, design scalable data pipelines, and implement robust processing with focus on performance and reliability. Strong Python expertise and familiarity with Gen AI are essential to translate business requirements into technical solutions.