Senior PySpark Lead Developer | Data Pipeline Architect

Luxoft; Województwo mazowieckie; Vor Ort; PLN 180.000 - 260.000; Vollzeit

Zusammenfassung

Luxoft in Poland is seeking a senior data engineer to lead the Surveillance Enhancement Program (SEP) focusing on PySpark data pipelines, MAR regulation compliance, and UI/workflow enhancements.

You will design architectures for large-scale data processing, optimize performance, and mentor a development team while collaborating with stakeholders. Knowledge of Python, Django, PostgreSQL, and Airflow is a plus.

Qualifikationen

• 15+ years of experience in data-intensive PySpark development and related tech.
• Experience leading a development team.
• Proven Python development expertise with OOP and exceptions.
• Strong DB connectivity, data loading and transformations.
• Solid Pandas/Numpy dataframes experience.
• Familiarity with Git-based workflows and SOA concepts.

Aufgaben

• Develop software applications based on business requirements.
• Maintain and enhance applications per project specs.
• Lead requirement analysis, design, development, testing and implementation.
• Propose new techniques and technologies for software development.
• Perform unit and user acceptance testing.
• Ensure deadlines and coding standards are met.
• Provide guidance to junior developers and conduct code reviews.
• Review business requirements and translate to reliable apps.
• Develop coding documentation and technical specs.
• Serve as primary development query contact.
• Analyze and resolve development issues.

Kenntnisse

Pyspark development

Window functions

Performance optimization

Spark architecture

Salting

Column pruning

Lead development team

Core Python

Gen AI understanding

DB connectivity

Pandas/Numpy

Git

Django

PostgreSQL

Apache Airflow

Jira

Tools

Git

Django

PostgreSQL

Apache Airflow

Jira

Jobbeschreibung

Luxoft in Poland is seeking a senior data engineer to lead the Surveillance Enhancement Program (SEP) focusing on PySpark data pipelines, MAR regulation compliance, and UI/workflow enhancements.

You will design architectures for large-scale data processing, optimize performance, and mentor a development team while collaborating with stakeholders. Knowledge of Python, Django, PostgreSQL, and Airflow is a plus.