Senior PySpark Developer

Luxoft Poland; Kraków; On-site; PLN 180,000 - 250,000; Full time

Job summary

Luxoft Poland is hiring for a senior backend engineer to design, develop, and maintain a large-scale Regulatory Reporting Platform. You will build data pipelines using Python and PySpark, handle ingestion, transformation, and enrichment from multiple internal sources, and ensure data quality and reliability across jurisdictions.

You will collaborate with analysts, QA, and cross-functional teams in a global agile setup, contributing to scalable and robust reporting solutions for regulatory

Qualifications

• 6+ years of commercial Python development experience.
• 2+ years of PySpark experience.
• Strong SQL skills and handling large datasets.

Responsibilities

• Design, develop, and maintain a large-scale Regulatory Reporting Platform.
• Build data processing pipelines using Python and PySpark.
• Implement scalable data ingestion, transformation, and enrichment from multiple sources.

Skills

Python

PySpark

SQL

Backend design

Distributed team

Job description

You will join a Regulatory Reporting Platform within a large US investment bank. The platform is responsible for collecting, processing, and transforming large volumes of data from multiple internal systems to produce regulatory reports for 240 jurisdictions worldwide. In total, the platform supports around 10,000 reports across a wide range of regulatory, compliance, financial, and risk reporting areas.

This is a large-scale, business-critical data platform operating in a highly regulated environment, where data accuracy, quality, traceability, and timely delivery are essential. The team works on building and enhancing data pipelines, implementing reporting logic, ensuring data validation and reconciliation, and supporting the generation of complex regulatory outputs.

The project is delivered by a global distributed team with colleagues located in Ukraine, Poland, and the US, offering close collaboration with engineers, analysts, and business stakeholders across multiple regions.

Responsibilities:

• Design, develop, and maintain solutions for a large-scale Regulatory Reporting Platform
• Build and enhance data processing pipelines using Java and Apache Spark
• Implement scalable logic for data ingestion, transformation, aggregation, and enrichment from multiple internal data sources
• Develop and optimize batch processing jobs to support the generation of regulatory reports across multiple jurisdictions
• Ensure high standards of data quality, validation, reconciliation, and consistency throughout the reporting lifecycle
• Translate business and regulatory requirements into robust technical solutions
• Collaborate with business analysts, QA engineers, and cross-functional teams to deliver reporting functionality
• Investigate and resolve data and application issues in development, test, and production environments
• Improve application performance, scalability, and reliability for high-volume data processing workloads
• Contribute to the design and implementation of reusable components and engineering best practices
• Participate in code reviews, testing, and release activities to ensure high-quality deliverables
• Work closely with distributed teams across Ukraine, Poland, and the US in an agile delivery model
• Support continuous improvement of the platform, including monitoring, troubleshooting, and operational stability

Mandatory Skills Description:

• 6+ years of commercial experience with Python
• 2+ years of PySpark experience
• Experience in designing and developing scalable, high-performance backend applications
• Good knowledge of SQL and working with large datasets from multiple data sources
• Strong problem-solving skills and ability to work effectively in a global distributed team

Nice-to-Have Skills Description:

• Experience in financial services, investment banking, or regulatory reporting projects
• Familiarity with data quality, reconciliation, and control frameworks
• Experience working with large-scale enterprise data platforms
• Knowledge of cloud environments and modern data platform architectures
• Understanding of CI/CD practices and automated deployment pipelines
• Experience working in Agile / Scrum teams within a distributed global setup