Databricks Developer with Python

Luxoft; Pune District; On-site; INR 2,500,000 - 4,500,000; Full time

Job summary

Luxoft in Pune, India is seeking a Senior Databricks Engineer to focus on Python development and lead the migration to Azure Databricks to boost performance, scalability, and efficiency.

You will develop, optimize, and automate regulatory reporting using PySpark and SQL, ensuring timely and accurate outputs.

Join a global agile team and translate business requirements into robust Databricks implementations with emphasis on reliability and data quality.

Qualifications

• Minimum of 4+ years of experience developing enterprise Databricks applications.
• Hands-on experience with Databricks ETL/data-loading mechanisms using Databricks Jobs, including relevant performance tuning and optimisation.
• Ability to independently understand business requirements and translate them into efficient, high-performing Databricks implementations.
• Data modelling experience would be an added advantage.
• Experience with Databricks deployments using Databricks Asset Bundles (DAB) would be an added advantage.

Responsibilities

• Implement high-quality solutions that address business needs and adhere to industry standards
• Drive automation of manual processes to the next level
• Closely collaborate within an agile team and a global agile organization
• Translate high-level business requirements into technical requirements addressing reliability, scalability, and performance needs

Skills

Python

Databricks

PySpark

SQL

ETL

Data modelling

Databricks Asset Bundles (DAB)

Tools

Azure Databricks

Databricks Jobs

Docker

Kubernetes

Git

GitHub

GitLab

Jenkins

Job description

We are looking for a Senior Databricks Engineer. The primary focus of this role is built in Python, while also leading the migration to Azure Databricks to improve performance, scalability, and efficiency.

This role involves developing, optimizing, and automating reporting processes using PySpark and SQL, ensuring accurate and timely regulatory report generation. This is an exciting opportunity to work on.

Responsibilities:

• Implement high-quality solutions that address business needs and adhere to industry standards
• Drive automation of manual processes to the next level
• Closely collaborate within an agile team and a global agile organization
• Translate high-level business requirements into technical requirements addressing reliability, scalability, and performance needs

Mandatory Skills Description:

• Minimum of 4+ years of experience developing enterprise Databricks applications.
• Hands-on experience with Databricks ETL/data-loading mechanisms using Databricks Jobs, including relevant performance tuning and optimisation.
• Ability to independently understand business requirements and translate them into efficient, high-performing Databricks implementations.
• Data modelling experience would be an added advantage.
• Experience with Databricks deployments using Databricks Asset Bundles (DAB) would be an added advantage.

Nice-to-Have Skills Description:

• ETL & Data Processing
• Financial Data Processing, experience handling risk, compliance, and regulatory data in an investment banking environment.
• Data Modeling, metadata management
• Git, GitHub, GitLab
• Jenkins
• Regulatory compliance knowledge: Basel, MiFID, GDPR
• Cloud security and access controls (IAM, RBAC)
• Familiarity with Docker, Kubernetes, Apache