Senior PySpark & Python Lead - Data Pipeline Architect

Luxoft; Warszawa; Vor Ort; PLN 400.000 - 640.000; Vollzeit

Zusammenfassung

Luxoft in Warsaw is seeking a senior data engineering lead to develop and optimize data-intensive PySpark applications. You will guide a development team, design scalable data pipelines, and ensure high-quality code and documentation.

The role requires strong Python expertise, experience with Pandas/Numpy, and knowledge of Spark architecture and performance tuning. Collaboration with cross-functional teams and adherence to standards are essential.

Qualifikationen

• 10+ years of experience in data intensive PySpark development with Spark architecture and optimization techniques.
• Experience leading a development team.
• Proven Python developer with strong Pandas/Numpy dataframes proficiency.
• Good understanding of Gen AI and data-driven applications.
• Strong DB connectivity, data loading, transformation and calculation skills.
• Experience with Pandas/Numpy dataframes, slicing, wrangling and aggregations.
• Familiarity with vector operations on dataframes/series and apply/map usage.
• Experience with concurrency and error handling in data pipelines (1–10 GB batches).
• Ability to translate business requirements into technical specs and design scalable data pipelines.
• Knowledge of Git and service-oriented architecture with potential integrations.

Aufgaben

• Develop software applications based on business requirements.
• Maintain software applications and implement enhancements per project specs.
• Participate in requirement analysis, design, development, testing, and deployment activities.
• Propose new techniques and technologies for software development.
• Perform unit testing and user acceptance testing to verify functionality.
• Complete assigned development tasks within deadlines.
• Adhere to coding standards and best practices.
• Provide guidance to junior developers when needed.
• Perform code reviews and recommend improvements.
• Review requirements and suggest changes to build reliable applications.
• Develop coding documentation and other technical specifications for projects.
• Act as a primary contact for development queries and concerns.
• Analyze and resolve development issues accurately.

Kenntnisse

PySpark data engineering

Spark architecture

Window functions

Data wrangling (Pandas/Numpy)

DB connectivity

Performance optimization

Salting

Column pruning

Team lead experience

Gen AI understanding

Data pipelines concurrency

Git version control

Python backend (Django)

PostgreSQL

Jobbeschreibung

Luxoft in Warsaw is seeking a senior data engineering lead to develop and optimize data-intensive PySpark applications. You will guide a development team, design scalable data pipelines, and ensure high-quality code and documentation.

The role requires strong Python expertise, experience with Pandas/Numpy, and knowledge of Spark architecture and performance tuning. Collaboration with cross-functional teams and adherence to standards are essential.