Senior Databricks Platform Engineer

Remotedxb; Dubai; On-site; AED 300,000 - 420,000; Full time

Job summary

GDIT seeks an experienced Senior Databricks Platform Architect to design, implement, and optimize the enterprise Lakehouse on Databricks within a UAE context. You will oversee workspaces, Unity Catalog, and scalable data architectures, while tuning Spark workloads and managing costs.

You will drive automation via Terraform and Databricks APIs, integrate with BI tools, and mentor data engineers across hybrid cloud environments using AWS or comparable platforms.

Qualifications

• MA/MS degree or equivalent with 12+ years of experience
• Hands-on experience implementing and administering the Databricks Lakehouse Platform
• Expertise in SQL, Spark performance tuning, and workload management
• Proficiency in Python and Bash for automation
• Experience with CI/CD pipelines (Jenkins, GitLab CI, GitHub Actions, Azure DevOps)
• Experience operating Databricks in AWS-based or hybrid cloud environments
• Familiarity with containerized environments like Docker and Kubernetes
• Knowledge of data ingestion, ETL/ELT, and distributed data processing
• Understanding of cloud security, IAM, and resource governance within Databricks and Unity Catalog

Responsibilities

• Design, configure, and maintain the enterprise Databricks Lakehouse Platform, including workspaces, Unity Catalog, and scalable data architectures
• Administer and optimize Databricks compute, clusters, SQL warehouses, and serverless capabilities
• Develop and optimize data pipelines, ETL/ELT processes, and ingestion frameworks using Apache Spark, Delta Lake, and Databricks Workflows
• Automate platform provisioning and deployments using Terraform, Python, and Databricks APIs
• Implement platform monitoring, logging, alerting, and observability
• Perform Spark and SQL performance tuning and troubleshooting
• Implement cost optimization strategies and DBU consumption forecasting
• Integrate Databricks with enterprise data sources, governance tools, and BI platforms
• Provide technical guidance and mentorship to data engineers and platform users

Skills

Databricks Lakehouse

SQL performance tuning

Python

CI/CD pipelines

Cloud environments

Education

MS/MA degree in a relevant field

Tools

Terraform

Databricks APIs

Docker/Kubernetes

Job description

Responsibilities

• Design, configure, and maintain the enterprise Databricks Lakehouse Platform, including workspaces, Unity Catalog, and scalable data architectures
• Administer and optimize Databricks compute, clusters, SQL warehouses, and serverless capabilities
• Develop and optimize data pipelines, ETL/ELT processes, and ingestion frameworks using Apache Spark, Delta Lake, and Databricks Workflows
• Automate platform provisioning and deployments using Terraform, Python, and Databricks APIs
• Implement platform monitoring, logging, alerting, and observability
• Perform Spark and SQL performance tuning and troubleshooting
• Implement cost optimization strategies and DBU consumption forecasting
• Integrate Databricks with enterprise data sources, governance tools, and BI platforms
• Provide technical guidance and mentorship to data engineers and platform users

Requirements

• MA/MS degree with 12+ years of experience (or equivalent experience in lieu of degree)
• Strong, hands-on experience implementing and administering the Databricks Lakehouse Platform
• Expertise in SQL, Spark performance tuning, and workload management
• Proficiency in Python and Bash for automation and scripting
• Experience with CI/CD pipelines (Jenkins, GitLab CI, GitHub Actions, Azure DevOps)
• Experience operating Databricks in AWS-based or hybrid cloud environments
• Familiarity with containerized environments like Docker and Kubernetes
• Knowledge of data ingestion, ETL/ELT, and distributed data processing
• Understanding of cloud security, IAM, and resource governance within Databricks and Unity Catalog

About the Company

GDIT is a global technology and professional services company that delivers technology solutions and mission services to every major agency across the U.S. government, defense and intelligence community.