Lead PySpark Developer - Data Platform & Surveillance

Luxoft; Województwo pomorskie; On-site; PLN 320,000 - 520,000; Full time

Job summary

Luxoft is seeking a senior data/pySpark engineer in Poland to drive data-intensive Spark-based platforms for regulatory surveillance. You will lead a development team, design scalable data pipelines, and implement performance optimizations while collaborating with stakeholders and ensuring code quality.

Responsibilities include full software lifecycle from analysis to deployment, applying best practices, and mentoring junior developers in a fast-paced environment with exposure to Gen AI

Qualifications

• 10+ years of data-intensive PySpark development and performance optimization.
• Leadership experience with a development team.
• Strong Python core development skills and OO design.
• Good understanding of Gen AI concepts and applications.
• Experience with classes, OOP, exception handling, and parallel processing.
• Solid DB connectivity, data loading, transformation and calculations.
• Extensive Pandas/Numpy dataframes experience including slicing and wrangling.
• Proficiency with Lambdas, Decorators and vectorized Pandas operations.
• Experience applying map/apply style functions and concurrency in pipelines.
• Ability to translate business requirements into technical specs and architecture.
• Working with Git and SOA-style integrations across services.
• Strong debugging skills and code review capabilities.

Responsibilities

• Develop software applications from business requirements.
• Maintain and enhance applications per project specs.
• Participate in requirements analysis, design, development, testing, implementation.
• Propose new techniques and technologies for software development.
• Perform unit testing and user acceptance testing.
• Ensure completion of development tasks within deadlines.
• Adhere to coding standards and best practices.
• Provide guidance to junior developers as needed.
• Conduct code reviews and recommend improvements.
• Review requirements and suggest changes for reliable applications.
• Create coding documentation and technical specs for projects.
• Act as primary contact for development queries.
• Analyze and resolve development issues accurately.

Skills

PySpark development

Team leadership

Python development

Gen AI

OOP concepts

DB connectivity

Pandas / NumPy

Lambdas / Decorators

Vector operations

Apply/Map functions

Data pipeline concurrency

Business requirement translation

Data pipeline architecture

Git / SOA integrations

Debugging

Tools

Django

PostgreSQL

Apache Airflow

Job description

Luxoft is seeking a senior data/pySpark engineer in Poland to drive data-intensive Spark-based platforms for regulatory surveillance. You will lead a development team, design scalable data pipelines, and implement performance optimizations while collaborating with stakeholders and ensuring code quality.

Responsibilities include full software lifecycle from analysis to deployment, applying best practices, and mentoring junior developers in a fast-paced environment with exposure to Gen AI