Senior Data Engineer (Databricks/Python/SQL)

Luxoft; Pune District; Hybrid; INR 3,000,000 - 5,000,000; Full time

Job summary

Luxoft in India seeks an experienced Data Engineer to advance regulatory data platforms. You will design and build scalable data pipelines, work with PySpark, Databricks, Delta Lake and Azure in a fast-paced, global team.

This role requires 8+ years in data engineering, strong Python/SQL skills, and experience with CI/CD, Power BI, and AI-assisted development. You will collaborate across engineering and product teams to deliver reliable, low-latency solutions.

Qualifications

• Bachelor's degree in Computer Science or a related IT field, with 8+ years of experience in a similar role.
• Advanced proficiency in Python and SQL.
• Hands-on experience building scalable batch and streaming data pipelines for near real-time analytics using PySpark (Spark SQL, Structured Streaming).
• Hands-on experience with Databricks, including Delta Lake and Delta Live Tables (DLT).
• Experience designing Data Lake / Lakehouse and Data Warehouse solutions, with a focus on NFRs such as high throughput and low latency.
• Experience with Azure cloud services, Unix/Linux, and shell scripting.
• Expertise in automated testing and CI/CD pipelines (GitLab CI or similar).
• Hands-on experience with Power BI.
• Practical use of AI tools in the SDLC (e.g. GitHub Copilot, AI-assisted coding and testing).

Responsibilities

• Design, implement, and maintain new and existing data-driven applications
• Ability to develop and see through solutions end-to-end.
• Engineer reliable data pipelines for sourcing, processing, distributing, and storing data
• Design data models that can support ever-growing volumes
• Contribute to all stages of the software development lifecycle
• Provide technical expertise, recommendations, and innovative solutions
• Work collaboratively alongside the product owner and other technologists
• Share expertise and best practices with colleagues and contribute regularly to our engineering culture

Skills

Python

SQL

PySpark

Spark SQL

Structured Streaming

Delta Lake

Delta Live Tables

Databricks

Azure

Unix/Linux

Shell scripting

CI/CD (GitLab CI)

Power BI

AI tools in SDLC

English (C1 Advanced)

Education

Bachelor's degree in Computer Science or related IT field

Tools

Databricks

Delta Lake

Delta Live Tables

Azure

GitLab CI

Power BI

Job description

Project Description:

You'll be working in the Regulatory Reporting team in Poland, in the Investment Bank within the Digital Operations Stream. Our purpose is to transform the way client does business by simplifying and improving the products and platforms relied on by our Regulatory and Data Clients. We play a crucial role in enabling the Investment Bank to meet its regulatory obligations and create value from its wealth of data. We develop and implement the most innovative technologies available within client in our solutions and products. Our small, autonomous Agile teams are empowered to respond to client demands while challenging the ways of working to improve our delivery experience. With members based in London, New York, Singapore, Zurich, Poland, and India, we pride ourselves on being an inclusive and diverse team. We are looking for experts keen to keep learning and changing the game with us. Diversity helps us grow together. That's why we are committed to fostering and advancing diversity, equity, and inclusion. It strengthens our business and brings value to our clients.

Responsibilities:

• Design, implement, and maintain new and existing data-driven applications
• Ability to develop and see through solutions end-to-end.
• Engineer reliable data pipelines for sourcing, processing, distributing, and storing data
• Design data models that can support ever-growing volumes
• Contribute to all stages of the software development lifecycle
• Provide technical expertise, recommendations, and innovative solutions
• Work collaboratively alongside the product owner and other technologists
• Share expertise and best practices with colleagues and contribute regularly to our engineering culture

Mandatory Skills Description:

• Bachelor's degree in Computer Science or a related IT field, with 8+ years of experience in a similar role.
• Advanced proficiency in Python and SQL.
• Hands-on experience building scalable batch and streaming data pipelines for near real-time analytics using PySpark (Spark SQL, Structured Streaming).
• Hands-on experience with Databricks, including Delta Lake and Delta Live Tables (DLT).
• Experience designing Data Lake / Lakehouse and Data Warehouse solutions, with a focus on NFRs such as high throughput and low latency.
• Experience with Azure cloud services, Unix/Linux, and shell scripting.
• Expertise in automated testing and CI/CD pipelines (GitLab CI or similar).
• Hands-on experience with Power BI.
• Practical use of AI tools in the SDLC (e.g. GitHub Copilot, AI-assisted coding and testing).

Nice-to-Have Skills Description:

• Experience in database development (e.g., SQL, PL/SQL, PostgreSQL, Azure SQL) is desirable

Languages:

• English: C1 Advanced