Databricks Platform Engineer & Automation Specialist

OpenTalent; United States; On-site; USD 140,000 - 190,000; Full time

Benefits offered by this job

401(k) matching

Dental insurance

Health insurance

Paid time off

Parental leave

Vision insurance

Job summary

OpenTalent is seeking a Databricks Platform Engineer to build, maintain, and optimize enterprise data lakehouse infrastructure. You will automate Databricks operations, ensure reliability, and enable data teams to leverage the platform efficiently.

The role focuses on platform provisioning, monitoring, cost optimization, and cross-functional collaboration to implement CI/CD and governance across cloud environments.

Qualifications

• 5+ years in platform engineering, cloud infrastructure, or IT operations with Databricks experience.
• Strong knowledge of Databricks administration including cluster setup and workspace management.
• Proficiency in Apache Spark for distributed data processing.
• Hands-on experience with Azure, AWS, or GCP GovCloud/Public Sector.
• Scripting/automation skills in Python, Bash, or PowerShell.
• Experience with IaC tools like Terraform, CloudFormation, or ARM templates.
• Solid understanding of cloud networking (VPCs, subnets, security groups).

Responsibilities

• Build and maintain Databricks workspace infrastructure across dev, staging, and prod.
• Design and operate Databricks platforms in cloud environments ensuring scalability and cost-efficiency.
• Manage and optimize Databricks clusters, including autoscaling and monitoring.
• Oversee deployment and configuration of Databricks workspaces with access controls.
• Collaborate to implement CI/CD pipelines for Databricks workflows and notebooks.
• Monitor performance, logging, and health; enforce data security and governance.
• Troubleshoot platform issues with cross-functional teams.
• Automate provisioning, upgrades, and scaling using IaC tools.
• Support data engineers and scientists integrating Databricks with data sources.

Skills

Platform engineering

Databricks administration

Apache Spark

Cloud platforms

Scripting/automation

IaC automation tools

Tools

Databricks

Terraform

CloudFormation

ARM templates

Job description

OpenTalent is seeking a Databricks Platform Engineer to build, maintain, and optimize enterprise data lakehouse infrastructure. You will automate Databricks operations, ensure reliability, and enable data teams to leverage the platform efficiently.

The role focuses on platform provisioning, monitoring, cost optimization, and cross-functional collaboration to implement CI/CD and governance across cloud environments.