Databricks Platform Engineer - Agility Technologies Inc

OpenTalent; United States; On-site; USD 140,000 - 190,000; Full time

Benefits offered by this job

401(k) matching

Dental insurance

Health insurance

Paid time off

Parental leave

Vision insurance

Job summary

OpenTalent is seeking a Databricks Platform Engineer to build, maintain, and optimize enterprise data lakehouse infrastructure. You will automate Databricks operations, ensure reliability, and enable data teams to leverage the platform efficiently.

The role focuses on platform provisioning, monitoring, cost optimization, and cross-functional collaboration to implement CI/CD and governance across cloud environments.

Qualifications

• 5+ years in platform engineering, cloud infrastructure, or IT operations with Databricks experience.
• Strong knowledge of Databricks administration including cluster setup and workspace management.
• Proficiency in Apache Spark for distributed data processing.
• Hands-on experience with Azure, AWS, or GCP GovCloud/Public Sector.
• Scripting/automation skills in Python, Bash, or PowerShell.
• Experience with IaC tools like Terraform, CloudFormation, or ARM templates.
• Solid understanding of cloud networking (VPCs, subnets, security groups).

Responsibilities

• Build and maintain Databricks workspace infrastructure across dev, staging, and prod.
• Design and operate Databricks platforms in cloud environments ensuring scalability and cost-efficiency.
• Manage and optimize Databricks clusters, including autoscaling and monitoring.
• Oversee deployment and configuration of Databricks workspaces with access controls.
• Collaborate to implement CI/CD pipelines for Databricks workflows and notebooks.
• Monitor performance, logging, and health; enforce data security and governance.
• Troubleshoot platform issues with cross-functional teams.
• Automate provisioning, upgrades, and scaling using IaC tools.
• Support data engineers and scientists integrating Databricks with data sources.

Skills

Platform engineering

Databricks administration

Apache Spark

Cloud platforms

Scripting/automation

IaC automation tools

Tools

Databricks

Terraform

CloudFormation

ARM templates

Job description

• 401(k) matching
• Dental insurance
• Health insurance
• Paid time off
• Parental leave
• Vision insurance

Role Description:

The Databricks Platform Engineer will build, maintain, and optimize the enterprise data lakehouse infrastructure. The ideal candidate will have strong DevOps and data engineering skills, with expertise in automating Databricks platform operations, ensuring reliability, and enabling data teams to efficiently leverage the platform. You will be responsible for platform provisioning, monitoring, cost optimization, and providing technical support for Databricks

workloads.

Key Responsibilities include

• Build and maintain Databricks workspace infrastructure across development, staging, and production environments
• Design, implement, and maintain Databricks platforms in cloud environments (Azure, AWS, or GCP), ensuring scalability, reliability, and cost-efficiency. Manage and optimize Databricks clusters, including autoscaling, resource allocation, monitoring, and troubleshooting.
• Oversee the deployment and configuration of Databricks workspaces, including user management, access controls, and workspace policies.
• Collaborate with infrastructure and DevOps teams to implement CI/CD pipelines for deploying Databricks workflows and notebooks. Monitor and manage performance, logging, and system health of the Databricks platform. Implement robust data security and governance measures in compliance with organizational policies and standards.
• Work with cross-functional teams to troubleshoot and resolve platform-related issues.
• Automate repetitive tasks, such as provisioning, upgrades, and scaling, using Infrastructure-as-Code (IaC) tools like Terraform or CloudFormation.
• Enable data engineers and data scientists by providing support for integrating Databricks with various data sources and tools.
• Stay up-to-date with the latest Databricks features, updates, and best practices to ensure the platform meets evolving business needs.

Minimum Skills:

• 5+ years of experience in platform engineering, cloud infrastructure, or IT operations, with 2+ years of hands-on experience managing Databricks environments.
• Strong knowledge of Databricks administration, including cluster setup, workspace management, and job orchestration.
• Proficiency in Apache Spark for distributed data processing.
• Hands-on experience with cloud platforms (Azure Government, AWS GovCloud, or GCP Public Sector).
• Proficiency in scripting and automation with Python, Bash, or PowerShell.
• Experience with Infrastructure-as-Code (IaC) tools like Terraform, CloudFormation, or ARM templates.
• Solid understanding of cloud networking concepts, such as VPCs, subnets, security groups, and private endpoints.