Senior PySpark & Data Platform Lead

Luxoft; Kraków; Hybrid; PLN 320,000 - 540,000; Full time

Job summary

Luxoft is seeking an experienced Senior Data Engineer in Kraków to drive the Surveillance Enhancement Program. You will build and optimize data-intensive PySpark pipelines, lead a development team, and translate business requirements into scalable technical solutions.

Expect collaboration across datasets, adherence to standards, and ongoing performance tuning. The role emphasizes complex data processing, DB connectivity, and modern Python development with OOP principles and Pandas/Numpy

Qualifications

• 10+ years of experience in data intensive PySpark development, Window Functions, performance optimization.
• Experience leading a development team.
• Proven experience as a Core Python developer.
• Good understanding of Gen AI.
• Experience developing classes, OOPs, exception handling, parallel processing.
• Strong knowledge of DB connectivity, data loading, transformation, calculation.
• Extensive experience in Pandas/Numpy dataframes, slicing, data wrangling, aggregations.
• Lambda functions and decorators.
• Vector operations on Pandas dataframes/series.
• Apply/map functions usage.
• Concurrency and error handling in data pipelines (1–10 GB batches).
• Ability to translate business requirements into technical specs.
• Data pipeline architecture design for concurrent processing.
• Experience in Git/SOA with integrations.
• Debugging applications.

Responsibilities

• Develop software applications based on business requirements.
• Maintain software applications and implement enhancements per project specs.
• Participate in requirement analysis, design, development, testing, and deployment.
• Propose new techniques and technologies for software development.
• Perform unit testing and user acceptance testing.
• Ensure tasks are completed within deadlines.
• Adhere to coding standards and best practices.
• Provide assistance to junior developers.
• Conduct code reviews and suggest improvements.
• Review requirements and recommend changes to ensure reliable applications.
• Develop coding documentation and technical specs for projects.
• Act as primary contact for development queries.
• Analyze and resolve development issues accurately.

Skills

Data-intensive PySpark

Window Functions

Performance Optimization

Spark Architecture

Salting

Column Pruning

Lead a Development Team

Core Python

Gen AI

OOPS / Exception handling

Parallel Processing

DB Connectivity

Pandas/Numpy

Lambda/Decorators

Vector Operations on Pandas

Apply/Map functions

Git

SOA & Integrations

Debugging

Tools

Git

Django

PostgreSQL

Apache Airflow

Job description

Luxoft is seeking an experienced Senior Data Engineer in Kraków to drive the Surveillance Enhancement Program. You will build and optimize data-intensive PySpark pipelines, lead a development team, and translate business requirements into scalable technical solutions.

Expect collaboration across datasets, adherence to standards, and ongoing performance tuning. The role emphasizes complex data processing, DB connectivity, and modern Python development with OOP principles and Pandas/Numpy