Remote Databricks Platform Engineer for Cloud Infra & ML

Piper Companies; United States; Remote; USD 150,000 - 180,000; Full time

Job summary

Zachary Piper is seeking a Databricks Infrastructure Engineer to design, build, and support a secure, scalable cloud data platform that enables enterprise analytics, forecasting, and machine learning workloads. You will administer Databricks workspaces, clusters, jobs, workflows, SQL warehouses, Unity Catalog, permissions, and service principals, implement IaC with Terraform, and build CI/CD pipelines.

This remote role requires collaboration with data engineers and IT, a strong foundation in

Qualifications

• Bachelor's degree or equivalent experience in Computer Science, Information Technology, Engineering, or a related field.
• 3+ years of experience supporting or engineering cloud data platforms.
• Hands-on Databricks administration experience.
• Experience with Azure, AWS, or Google Cloud Platform.
• Experience with Terraform or other Infrastructure as Code tools.
• Experience building and supporting CI/CD pipelines.
• Strong Python and SQL skills.
• Knowledge of data engineering concepts including data lakes, batch processing, streaming pipelines, and data quality controls.
• Understanding of cloud networking, identity management, monitoring, logging, and security best practices.

Responsibilities

• Design, deploy, and support Databricks platform infrastructure and cloud-based data environments.
• Administer Databricks workspaces, clusters, jobs, workflows, SQL warehouses, Unity Catalog, permissions, and service principals.
• Implement Infrastructure as Code using Terraform and related automation tools.
• Build and maintain CI/CD pipelines for Databricks notebooks, workflows, and data solutions.
• Partner with Data Engineers, Data Scientists, and IT teams to deliver reliable analytics and machine learning platforms.
• Monitor platform performance, reliability, usage, and costs.
• Optimize cluster configurations, workload scheduling, storage, and compute resources.
• Implement security controls including IAM, access governance, auditing, logging, and secrets management.
• Troubleshoot production issues, perform root cause analysis, and support disaster recovery planning.
• Create operational documentation, runbooks, and platform standards.
• Evaluate and implement new cloud and Databricks capabilities to improve performance and scalability.

Skills

Databricks admin

Python

SQL

CI/CD pipelines

Terraform

Cloud platforms

Documentation

Communication

Education

Bachelor's degree or equivalent experience

Tools

Azure

AWS

GCP

Terraform

GitHub Actions

Unity Catalog

Job description

Zachary Piper is seeking a Databricks Infrastructure Engineer to design, build, and support a secure, scalable cloud data platform that enables enterprise analytics, forecasting, and machine learning workloads. You will administer Databricks workspaces, clusters, jobs, workflows, SQL warehouses, Unity Catalog, permissions, and service principals, implement IaC with Terraform, and build CI/CD pipelines.

This remote role requires collaboration with data engineers and IT, a strong foundation in