Senior Databricks Platform Architect & Engineer

Luxoft; Abu Dhabi; On-site; AED 120,000 - 200,000; Full time

Job summary

A technology consulting firm in Abu Dhabi seeks a skilled DataBricks Platform Engineer to manage and optimize data systems. The ideal candidate will have over 10 years in IT and a deep understanding of data engineering, alongside proficiency in Python and PySpark. Key responsibilities include ensuring system performance and security, optimizing ETL processes, and providing insights into platform usage. This position offers an exciting opportunity to work at the intersection of data platforms and software engineering.

Qualifications

• Minimum 10 Years of experience in IT/Data.
• Minimum 3 years of experience as a Databricks Data Platform Engineer.
• Expertise in storage, networking, and compute on Azure or AWS.

Responsibilities

• Manage and optimize Databricks data platform.
• Ensure high availability, security, and performance of data systems.
• Provide valuable insights about data platform usage.
• Optimize computing and storage for large-scale data processing.
• Design and maintain system libraries (Python) used in ETL pipelines and platform governance.

Skills

Databricks Data Platform Engineering

PySpark

Python for ETL

SQL Optimization

Data Warehousing

ETL Processes

Data Quality Tools

Monitoring Tools

Git

Cloud (Azure, AWS)

Education

Bachelor's in IT or related field

Tools

Docker

Kubernetes

Terraform

Job description

A technology consulting firm in Abu Dhabi seeks a skilled DataBricks Platform Engineer to manage and optimize data systems. The ideal candidate will have over 10 years in IT and a deep understanding of data engineering, alongside proficiency in Python and PySpark. Key responsibilities include ensuring system performance and security, optimizing ETL processes, and providing insights into platform usage. This position offers an exciting opportunity to work at the intersection of data platforms and software engineering.