Senior PySpark Lead & Data Pipeline Architect

Luxoft; Gdańsk; Vor Ort; PLN 180.000 - 300.000; Vollzeit

Benefits dieser Stelle

Private Medical care

Dental care

Life Insurance

Internal Mobility program

Zusammenfassung

Luxoft in Poland is seeking a senior data engineering professional to lead the development of the Surveillance Enhancement Program (SEP). You will design and optimize PySpark-based data pipelines, integrate Gen AI capabilities, and collaborate with cross-functional teams across projects and locations.

The role requires 10+ years in data-intensive PySpark development, strong Python skills, and experience mentoring developers.

Qualifikationen

• 10+ years of experience in data-intensive PySpark development and Spark architecture.
• Experience leading a development team.
• Proven experience as a core Python developer.
• Strong knowledge of DB connectivity, data loading, transformation, calculation.
• Extensive experience with Pandas and NumPy dataframes.
• Knowledge of Gen AI.
• Concurrency and error handling in data pipelines (1-10 GB batches).
• Ability to understand business requirements and translate them into technical requirements.
• Experience with Git and service-oriented architecture and integrations.

Aufgaben

• Develop software applications based on business requirements.
• Maintain software applications and make enhancements according to project specifications.
• Participate in requirement analysis, design, development, testing, and implementation activities.
• Propose new techniques and technologies for software development.
• Perform unit testing and user acceptance testing to evaluate application functionality.
• Ensure to complete the assigned development tasks within the deadlines.
• Work in compliance with coding standards and best practices.
• Provide assistance to Junior Developers when needed.
• Perform code reviews and recommend improvements.
• Review business requirements and recommend changes to develop reliable applications.
• Develop coding documentation and other technical specifications for assigned projects.
• Act as primary contact for development queries and concerns.
• Analyze and resolve development issues accurately.

Kenntnisse

PySpark development

Spark architecture

Window functions

Performance optimization

Salting

Column pruning

Lead a development team

Core Python

Gen AI

DB connectivity

Pandas

NumPy

Dataframes

Vector operations

Concurrency

Apply/map functions

Git

SOA architecture

Integrations

Requirements translation

Tools

Django

PostgreSQL

Jobbeschreibung

Luxoft in Poland is seeking a senior data engineering professional to lead the development of the Surveillance Enhancement Program (SEP). You will design and optimize PySpark-based data pipelines, integrate Gen AI capabilities, and collaborate with cross-functional teams across projects and locations.

The role requires 10+ years in data-intensive PySpark development, strong Python skills, and experience mentoring developers.