Senior Databricks Data Platform Engineer

Luxoft; Abu Dhabi; On-site; AED 300,000 - 420,000; Full time

Job summary

Luxoft is seeking a Databricks Platform Engineer to help evolve the Lakehouse platform. You will design shared capabilities, build internal libraries, and implement governance and integration patterns at scale.

You will collaborate with stakeholders, write production-grade Python, and ensure reliability via tests, incident handling, and observability integrations across the enterprise.

Qualifications

• Over 10 years in IT/Data
• Minimum 3 years Databricks platform enablement
• 3+ years designing and maintaining APIs for automation, serverless workflows, release management
• Advanced Python for production-grade libraries/services
• Solid distributed processing concepts (PySpark beneficial)
• Experience designing and integrating REST APIs
• Advanced SQL for troubleshooting and performance (Oracle, MS SQL)
• Experience with Git and CI/CD workflows
• Familiarity with monitoring/observability tools
• Knowledge of scheduling/orchestration tools (Airflow, Stonebranch, Control-M)
• Understanding of data quality/governance (Great Expectations)
• Strong cloud infra fundamentals (DNS, identity, load balancing)
• English: C1 Advanced
• Seniority: Lead

Responsibilities

• Platform Capability Engineering: design and implement scalable platform solutions.
• Develop reusable Python libraries, SDKs, and internal frameworks.
• Expose platform capabilities to support data products strategy.
• Build automation for provisioning, governance, config management, and standards.

Skills

Databricks platform engineering

Python production-grade

APIs design & integration

CI/CD workflows

Observability tooling

Security & governance

Cloud infrastructure

REST APIs

PySpark knowledge

Leadership

English (Advanced)

Tools

Git

CI/CD

ServiceNow

Prometheus

Grafana

Dynatrace

Job description

Job description

Project descriptionWe are seeking a highly skilled and motivated Databricks Platform Engineer to join its core Lakehouse Platform team.This role focuses on engineering and evolving the Databricks-based Lakehouse platform itself, building advanced platform capabilities, reusable services, automation frameworks, and enterprise integrations that enable ADIA’s Data & AI ecosystem at scale.While the platform supports data pipelines and analytics workloads, this is not a traditional data engineering role focused on building ETL jobs or delivering datasets. Instead, the role is centred on:

• Designing and developing shared platform capabilities
• Building internal libraries and APIs
• Implementing governance, security, and integration patterns
• Ensuring scalability, reliability, and operational excellence

The ideal candidate combines strong Python software engineering expertise with solid infrastructure understanding and experience integrating enterprise systems. Clear, precise, and articulate communication is essential.The primary focus of this role is building and enhancing platform capabilities, not developing business ETL pipelines or reporting datasets.

Responsibilities

• Platform Capability Engineering
  • Collaborate with stakeholders during requirements clarification and sprint planning to ensure alignment with platform strategy and enterprise standards.
  • Design and implement technical solutions on ADIA’s Lakehouse platform (Databricks), including:
  • Prototyping and productising new platform capabilities.
  • Developing reusable Python libraries, SDKs, and internal frameworks.
  • Exposing platform capabilities to support ADIA’s Data Products strategy and broader Data & AI ecosystem.
  • Build automation for provisioning, governance controls, configuration management, and operational standards.

Enterprise Integration & APIs

• Integrate the Lakehouse platform with enterprise systems, including:
• Incident and monitoring systems (e.g., ServiceNow).
• Identity and access management solutions.
• Observability platforms (e.g., Dynatrace, Prometheus, Grafana).
• Design and implement secure API-based integrations using enterprise authentication and authorisation mechanisms.

Reliability, Quality & Operations

• Develop and maintain unit and integration tests to ensure quality and resilience of platform components.
• Support QA teams during acceptance testing of platform features.
• Act as a third-line engineer for complex production incidents affecting platform stability.
• Collaborate with cloud and infrastructure teams to deliver secure, scalable, and reliable platform architecture.

Skills

Must have

• Minimum 10 Years of experience in IT/Data.
• Minimum 3 years of expertise in Databricks from a platform enablement perspective.
• 3+ years of experience in designing, writing, and maintaining APIs used for Platform automation, Serverless workflows, Deployment pipelines, release management and repository management
• Advanced Python skills for production-grade library and service development.
• Solid understanding of distributed processing concepts (PySpark knowledge beneficial, but not limited to ETL use cases).
• Experience designing and integrating REST APIs.
• Advanced SQL skills for troubleshooting and performance optimisation (e.g., Oracle, MS SQL).
• Experience with Git and CI/CD workflows.
• Familiarity with monitoring and observability tools (e.g., ServiceNow, Prometheus, Grafana).
• Knowledge of scheduling and orchestration tools (e.g., Stonebranch, Control-M, Airflow).
• Understanding of data quality and governance frameworks (e.g., Great Expectations).
• Strong grasp of cloud infrastructure fundamentals:DNSCertificatesIdentity & access managementNetworking conceptsLoad balancing
• OtherLanguagesEnglish: C1 AdvancedSeniorityLead

Nice to have

• Containerization & Orchestration: Docker, Kubernetes.
• Infrastructure as Code (IaC): Terraform.
• Understanding of Investment Data domain (desired).