Senior Databricks Engineer

Remotedxb; Dubai; On-site; AED 250,000 - 400,000; Full time

Job summary

Geisinger is seeking a seasoned Databricks and data engineering lead to manage workspaces, clusters, jobs and security, ensuring platform health across our cloud ecosystem. You will build data pipelines with PySpark and T-SQL, and create automation tooling using Python, Bash and PowerShell, while enforcing data governance and security best practices.

You'll translate ideas into architectures and actionable tasks in Azure DevOps, participate in 24x7 on-call rotations for Databricks and SQL

Qualifications

• Extensive Databricks administration experience.
• Proficient in Python, Bash and PowerShell.
• Experience with PySpark and T-SQL data processing.
• Solid knowledge of data governance and security.
• Minimum 12 years of relevant experience.
• At least 3 years of people, process, or project leadership.

Responsibilities

• Manage Databricks workspaces, clusters, jobs, and security.
• Build data pipelines and transformations with PySpark and T-SQL.
• Create automation tooling using Python, Bash, and PowerShell.
• Implement data governance and access controls.
• Contribute to design architectures and task planning in Azure DevOps.
• Support Databricks and SQL Server in a 24x7 on-call rotation.
• Produce documentation including design specs and test plans.

Skills

Databricks admin

Python

Bash

PowerShell

PySpark

T-SQL

Data governance

Information security

People management

Project management

Tools

Terraform

Azure DevOps

GitHub

AWS Glue

AWS Lambda

MS SQL Server

Job description

Responsibilities

• Manage Databricks workspaces, clusters, jobs, security configurations, and overall platform health
• Build and optimize data pipelines and transformations using PySpark and T-SQL
• Create automation and tooling using Python, Bash, and PowerShell
• Implement and automate data governance, access controls, and compliance best practices
• Translate conversational ideas into clear solution architectures and actionable tasks in Azure DevOps
• Participate in a 24x7x365 on-call rotation to support Databricks and SQL Server
• Produce high-quality documentation including design specs, build docs, and test plans

Requirements

• Solid experience with Databricks administration
• Strong proficiency in Python, Bash, and PowerShell
• Experience with PySpark and/or T-SQL for data processing
• Strong understanding of data governance and information security best practices
• Minimum of 12 years of relevant experience
• Minimum of 3 years experience managing people, processes, or projects

Preferred Qualifications

• Familiarity with Terraform for infrastructure as code
• Experience creating AI/BI dashboards
• Knowledge of CI/CD pipelines using Azure DevOps or GitHub
• Hands-on experience with AWS data ingestion tools like Glue and Lambda
• Experience with MS SQL Server administration

About the Company

Geisinger is a healthcare system serving more than 1 million people, comprising nine hospital campuses, a health plan, two research centers, and the Geisinger Commonwealth School of Medicine.