Databricks Developer with Python

Luxoft; Pune District; On-site; INR 1,500,000 - 2,500,000; Full time

Job summary

Luxoft in Pune, India is seeking a Senior Databricks Engineer to advance Python-based development and lead the migration to Azure Databricks, improving performance, scalability, and efficiency.

The role involves building, optimizing, and automating reporting with PySpark and SQL, in an agile, global environment.

Qualifications

• Minimum 6+ years of hands-on application development experience using Databricks with Python.
• Experience with PySpark, Pandas, NumPy, and unit testing.
• Hands-on experience with SQL Server/ PostgreSQL database.
• Practical experience with microservice architectures, CI/CD pipelines (preferably with GitLab).
• Good to have exposure to cloud native services on Azure Experience in translating high-level business requirements into IT requirements.

Responsibilities

• Implement high-quality solutions that address business needs and adhere to industry standards.
• Drive automation of manual processes to the next level.
• Closely collaborate within an agile team and a global agile organization.
• Translate high-level business requirements into technical requirements addressing reliability, scalability, and performance needs.

Skills

Python

Databricks

PySpark

Pandas

NumPy

SQL Server

PostgreSQL

CI/CD

GitLab

Git

Docker

Kubernetes

Azure

Azure Databricks

Tools

SQL Server

PostgreSQL

Docker

Kubernetes

Azure

Apache

Job description

We are looking for a Senior Databricks Engineer. The primary focus of this role is built in Python, while also leading the migration to Azure Databricks to improve performance, scalability, and efficiency.

This role involves developing, optimizing, and automating reporting processes using PySpark and SQL, ensuring accurate and timely regulatory report generation. This is an exciting opportunity to work on.

Responsibilities:

• Implement high-quality solutions that address business needs and adhere to industry standards
• Drive automation of manual processes to the next level
• Closely collaborate within an agile team and a global agile organization
• Translate high-level business requirements into technical requirements addressing reliability, scalability, and performance needs

Mandatory Skills Description:

• Minimum 6+ years of hands-on application development experience using Databricks with Python
• Experience with PySpark, Pandas, NumPy, and unit testing
• Hands-on experience with SQL Server/ PostgreSQL database
• Practical experience with microservice architectures, CI/CD pipelines (preferably with GitLab)
• Good to have exposure to cloud native services on Azure Experience in translating high-level business requirements into IT requirements

Nice-to-Have Skills Description:

• ETL & Data Processing
• Financial Data Processing, experience handling risk, compliance, and regulatory data in an investment banking environment.
• Data Modeling, metadata management
• Git, GitHub, GitLab
• Jenkins
• Regulatory compliance knowledge: Basel, MiFID, GDPR
• Cloud security and access controls (IAM, RBAC)
• Familiarity with Docker, Kubernetes, Apache