Senior Databricks Data Platform Engineer

Luxoft; Abu Dhabi; On-site; AED 350,000 - 650,000; Full time

Job summary

Luxoft in Abu Dhabi is seeking a Databricks Platform Engineer to advance the Lakehouse platform, build reusable libraries, and drive enterprise integrations at scale.

The role emphasizes platform capabilities over traditional ETL work, requiring strong Python expertise, API design, and cloud-infra proficiency. Expect collaboration across architecture, security, and operations teams.

Qualifications

• 10+ years IT/Data experience.
• 3+ years Databricks platform enablement experience.
• Experience designing and exposing REST APIs.
• Advanced Python for production-grade libraries.
• Strong knowledge of SDLC, CI/CD, and version control.
• Familiarity with monitoring/observability tools.
• Understanding of cloud infrastructure fundamentals.

Responsibilities

• Platform capability engineering and prototyping.
• Develop reusable Python libraries, SDKs and frameworks.
• Expose platform capabilities to support data products strategy.
• Build automation for provisioning, governance, and configuration management.
• Integrate Lakehouse with enterprise systems (ServiceNow, IAM, observability).
• Design secure API-based integrations with enterprise authentication.
• Develop unit/integration tests; support QA; act on production incidents.
• Collaborate with cloud and infra teams for scalable architecture.

Skills

Databricks platform engineering

Python production-grade libraries

API design & development

REST APIs integration

CI/CD workflows

Observability tools

Security & governance patterns

Cloud infrastructure fundamentals

Distributed processing (PySpark)

Automation and scripting

Git

Tools

Docker

Kubernetes

Job description

Project description

We are seeking a highly skilled and motivated Databricks Platform Engineer to join its core Lakehouse Platform team.This role focuses on engineering and evolving the Databricks-based Lakehouse platform itself, building advanced platform capabilities, reusable services, automation frameworks, and enterprise integrations that enable ADIA’s Data & AI ecosystem at scale.While the platform supports data pipelines and analytics workloads, this is not a traditional data engineering role focused on building ETL jobs or delivering datasets. Instead, the role is centred on:Designing and developing shared platform capabilitiesBuilding internal libraries and APIsImplementing governance, security, and integration patternsEnsuring scalability, reliability, and operational excellenceThe ideal candidate combines strong Python software engineering expertise with solid infrastructure understanding and experience integrating enterprise systems. Clear, precise, and articulate communication is essential.The primary focus of this role is building and enhancing platform capabilities, not developing business ETL pipelines or reporting datasets.

Responsibilities

• Platform Capability Engineering
• Collaborate with stakeholders during requirements clarification and sprint planning to ensure alignment with platform strategy and enterprise standards.
• Design and implement technical solutions on ADIA’s Lakehouse platform (Databricks), including:
  • Prototyping and productising new platform capabilities
  • Developing reusable Python libraries, SDKs, and internal frameworks
  • Exposing platform capabilities to support ADIA’s Data Products strategy and broader Data & AI ecosystem.
• Build automation for provisioning, governance controls, configuration management, and operational standards.Enterprise Integration & APIs
• Integrate the Lakehouse platform with enterprise systems, including:
  • Incident and monitoring systems (e.g., ServiceNow).
  • Identity and access management solutions
  • Observability platforms (e.g., Dynatrace, Prometheus, Grafana)
• Design and implement secure API-based integrations using enterprise authentication and authorisation mechanisms.Reliability, Quality & Operations
• Develop and maintain unit and integration tests to ensure quality and resilience of platform components.
• Support QA teams during acceptance testing of platform features.
• Act as a third-line engineer for complex production incidents affecting platform stability.
• Collaborate with cloud and infrastructure teams to deliver secure, scalable, and reliable platform architecture.

SKILLS

Must have

• Minimum 10 Years of experience in IT/Data.
• Minimum 3 years of expertise in Databricks from a platform enablement perspective.
• 3+ years of experience in designing, writing, and maintaining APIs used for Platform automation, Serverless workflows, Deployment pipelines, release management and repository management
• Advanced Python skills for production-grade library and service development.
• Solid understanding of distributed processing concepts (PySpark knowledge beneficial, but not limited to ETL use cases).
• Experience designing and integrating REST APIs.
• Advanced SQL skills for troubleshooting and performance optimisation (e.g., Oracle, MS SQL).
• Experience with Git and CI/CD workflows.
• Familiarity with monitoring and observability tools (e.g., ServiceNow, Prometheus, Grafana).
• Knowledge of scheduling and orchestration tools (e.g., Stonebranch, Control-M, Airflow).
• Understanding of data quality and governance frameworks (e.g., Great Expectations).
• Strong grasp of cloud infrastructure fundamentals:o DNSo Certificateso Identity & access managemento Networking conceptso Load balancing

Nice to have

• Containerization & Orchestration: Docker, Kubernetes.
• Infrastructure as Code (IaC): Terraform.
• Understanding of Investment Data domain (desired).