Databricks Infrastructure Engineer

Piper Companies; United States; Remote; USD 150,000 - 180,000; Full time

Job summary

Zachary Piper is seeking a Databricks Infrastructure Engineer to design, build, and support a secure, scalable cloud data platform that enables enterprise analytics, forecasting, and machine learning workloads. You will administer Databricks workspaces, clusters, jobs, workflows, SQL warehouses, Unity Catalog, permissions, and service principals, implement IaC with Terraform, and build CI/CD pipelines.

This remote role requires collaboration with data engineers and IT, a strong foundation in

Qualifications

• Bachelor's degree or equivalent experience in Computer Science, Information Technology, Engineering, or a related field.
• 3+ years of experience supporting or engineering cloud data platforms.
• Hands-on Databricks administration experience.
• Experience with Azure, AWS, or Google Cloud Platform.
• Experience with Terraform or other Infrastructure as Code tools.
• Experience building and supporting CI/CD pipelines.
• Strong Python and SQL skills.
• Knowledge of data engineering concepts including data lakes, batch processing, streaming pipelines, and data quality controls.
• Understanding of cloud networking, identity management, monitoring, logging, and security best practices.

Responsibilities

• Design, deploy, and support Databricks platform infrastructure and cloud-based data environments.
• Administer Databricks workspaces, clusters, jobs, workflows, SQL warehouses, Unity Catalog, permissions, and service principals.
• Implement Infrastructure as Code using Terraform and related automation tools.
• Build and maintain CI/CD pipelines for Databricks notebooks, workflows, and data solutions.
• Partner with Data Engineers, Data Scientists, and IT teams to deliver reliable analytics and machine learning platforms.
• Monitor platform performance, reliability, usage, and costs.
• Optimize cluster configurations, workload scheduling, storage, and compute resources.
• Implement security controls including IAM, access governance, auditing, logging, and secrets management.
• Troubleshoot production issues, perform root cause analysis, and support disaster recovery planning.
• Create operational documentation, runbooks, and platform standards.
• Evaluate and implement new cloud and Databricks capabilities to improve performance and scalability.

Skills

Databricks admin

Python

SQL

CI/CD pipelines

Terraform

Cloud platforms

Documentation

Communication

Education

Bachelor's degree or equivalent experience

Tools

Azure

AWS

GCP

Terraform

GitHub Actions

Unity Catalog

Job description

Overview

Zachary Piper is seeking a Databricks Infrastructure Engineer to design, build, and support a secure, scalable cloud data platform that enables enterprise analytics, forecasting, and machine learning workloads. This role focuses on Databricks administration, cloud infrastructure, automation, platform reliability, security, and cost optimization.

Responsibilities

• Design, deploy, and support Databricks platform infrastructure and cloud-based data environments.
• Administer Databricks workspaces, clusters, jobs, workflows, SQL warehouses, Unity Catalog, permissions, and service principals.
• Implement Infrastructure as Code using Terraform and related automation tools.
• Build and maintain CI/CD pipelines for Databricks notebooks, workflows, and data solutions.
• Partner with Data Engineers, Data Scientists, and IT teams to deliver reliable analytics and machine learning platforms.
• Monitor platform performance, reliability, usage, and costs.
• Optimize cluster configurations, workload scheduling, storage, and compute resources.
• Implement security controls including IAM, access governance, auditing, logging, and secrets management.
• Troubleshoot production issues, perform root cause analysis, and support disaster recovery planning.
• Create operational documentation, runbooks, and platform standards.
• Evaluate and implement new cloud and Databricks capabilities to improve performance and scalability.

Required Qualifications

• Bachelor's degree or equivalent experience in Computer Science, Information Technology, Engineering, or a related field.
• 3+ years of experience supporting or engineering cloud data platforms.
• Hands‑on Databricks administration experience.
• Experience with Azure, AWS, or Google Cloud Platform.
• Experience with Terraform or other Infrastructure as Code tools.
• Experience building and supporting CI/CD pipelines.
• Strong Python and SQL skills.
• Knowledge of data engineering concepts including data lakes, batch processing, streaming pipelines, and data quality controls.
• Understanding of cloud networking, identity management, monitoring, logging, and security best practices.
• Strong troubleshooting, communication, and documentation skills.

Preferred Qualifications

• Experience with Unity Catalog, MLflow, Delta Lake, Delta Live Tables, Databricks Asset Bundles, or model serving.
• Experience supporting machine learning, forecasting, analytics, supply chain, or demand planning environments.
• Databricks or cloud platform certifications.
• Experience with cloud cost optimization and FinOps practices.
• Familiarity with ERP, SAP, supply chain, sales, inventory, or related data integrations.

Compensation

Salary: $150,000 - $180,000

Location: REMOTE

Key Skills

Databricks, Azure, AWS, GCP, Terraform, Infrastructure as Code, CI/CD, Azure DevOps, GitHub Actions, Python, SQL, Unity Catalog, MLflow, Delta Lake, Delta Live Tables, Data Engineering, Cloud Infrastructure, Platform Engineering, IAM, Monitoring, Logging, FinOps, Data Lakes, Machine Learning Infrastructure.