Remote Databricks Infrastructure Engineer

Zachary Piper, LLC; Zachary (LA); Remote; USD 115,000 - 130,000; Full time

Benefits offered by this job

Medical benefits

Dental benefits

Vision benefits

401(k)

PTO

Paid Holidays

Job summary

Piper Companies is seeking a Databricks Infrastructure Engineer to design, build, and manage enterprise cloud data infrastructure supporting forecasting, analytics, and ML initiatives.

The role focuses on administering Databricks environments, implementing automation, securing platforms, and optimizing data pipelines for reliability and cost efficiency across Azure/AWS/GCP environments.

Qualifications

• Bachelor's degree in Computer Science, Information Technology, Engineering, or related field or equivalent experience.
• 3+ years in cloud data platform engineering, infrastructure administration, or Databricks platform management.
• Hands-on Databricks administration: clusters, jobs, workflows, SQL warehouses, Unity Catalog, workspace configs.
• Experience with Azure, AWS, or Google Cloud; IaC using Terraform; Git-based deployment workflows.
• Proficiency in Python, SQL, scripting; strong data engineering knowledge and security controls.

Responsibilities

• Design, configure, and maintain Databricks platform infrastructure for analytics and forecasting workloads.
• Manage Databricks workspaces, clusters, SQL warehouses, jobs, and Unity Catalog access controls.
• Develop IaC solutions with Terraform and cloud automation; implement CI/CD for notebooks and pipelines.
• Support data ingestion, transformation, feature engineering, and ML workloads in the cloud.
• Monitor performance, optimize costs, and enforce security, IAM, and data governance.

Skills

Python

SQL

Scripting

Terraform

CI/CD

Cloud computing

Data governance

IAM

Education

Bachelor's degree in CS/IT/Engineering

Tools

Terraform

GitHub Actions

Azure DevOps

Databricks

CI/CD pipelines

Job description

Piper Companies is seeking a Databricks Infrastructure Engineer to support the design, development, and administration of enterprise cloud data infrastructure supporting forecasting, analytics, and machine learning initiatives. The Databricks Infrastructure Engineer will be responsible for managing Databricks environments, implementing infrastructure automation, optimizing data pipelines, and ensuring secure, scalable, and reliable cloud platform operations.

Responsibilities of the Databricks Infrastructure Engineer include:

• Design, configure, administer, and maintain Databricks platform infrastructure supporting enterprise data engineering, analytics, and forecasting workloads
• Manage Databricks workspaces, clusters, SQL warehouses, jobs, workflows, Unity Catalog, and access controls
• Develop and maintain Infrastructure as Code (IaC) solutions using Terraform and cloud automation technologies
• Implement CI/CD pipelines for Databricks notebooks, workflows, data pipelines, and platform deployments
• Support cloud-based data ingestion, transformation, feature engineering, and machine learning workloads
• Monitor platform performance, pipeline reliability, compute utilization, and cloud infrastructure costs
• Optimize Databricks clusters, workload scheduling, storage configurations, and resource allocation
• Implement security controls, including identity and access management, network security, secrets management, and data governance
• Troubleshoot infrastructure issues, pipeline failures, and performance bottlenecks across Databricks environments
• Collaborate with Data Engineers, Data Scientists, DevOps Engineers, and Enterprise Architecture teams to support scalable data solutions
• Support incident response, root‑cause analysis, disaster recovery, and business continuity initiatives
• Develop technical documentation, operational procedures, and platform engineering best practices

Qualifications for the Databricks Infrastructure Engineer include:

• Bachelor's degree in Computer Science, Information Technology, Engineering, or a related discipline, or equivalent professional experience
• 3+ years of experience in cloud data platform engineering, infrastructure administration, or Databricks platform management
• Strong hands‑on experience administering Databricks environments, including clusters, jobs, workflows, SQL warehouses, and workspace configurations
• Experience with cloud platforms such as Microsoft Azure, AWS, or Google Cloud Platform
• Experience implementing Infrastructure as Code using Terraform or similar technologies
• Hands‑on experience with CI/CD pipelines, Git‑based source control, and automated deployment workflows
• Proficiency with Python, SQL, and scripting for infrastructure automation and platform management
• Understanding of data engineering concepts, including ETL/ELT pipelines, batch and streaming data processing, Delta Lake, and data quality
• Experience with cloud networking, identity and access management, security controls, monitoring, and logging
• Strong troubleshooting, technical documentation, communication, and cross‑functional collaboration skills

Preferred Qualifications:

• Experience with Databricks Unity Catalog, MLflow, Delta Live Tables, Databricks Asset Bundles, and model serving
• Experience supporting forecasting, machine learning, supply chain, demand planning, or enterprise analytics environments
• Databricks Certified Data Engineer, Databricks Certified Platform Administrator, or relevant cloud infrastructure certifications
• Experience with cloud cost optimization, FinOps practices, and Databricks compute optimization
• Familiarity with SAP, ERP systems, supply chain data, inventory management, or sales forecasting integrations
• Experience with Azure DevOps, GitHub Actions, and automated Databricks deployment frameworks

Compensation for the Databricks Infrastructure Engineer includes:

• Salary: $115,000 - $130,000
• Comprehensive Benefits Package including Medical, Dental, Vision, 401(k), PTO, and Paid Holidays

Keywords:

Databricks Infrastructure Engineer, Databricks Engineer, Databricks Platform Engineer, Databricks Administrator, Cloud Data Engineer, Cloud Infrastructure Engineer, Data Platform Engineer, Databricks, Azure Databricks, Databricks Workspaces, Databricks Clusters, Databricks SQL, SQL Warehouses, Databricks Workflows, Databricks Jobs, Unity Catalog, Delta Lake, Delta Live Tables, MLflow, Databricks Asset Bundles, Data Engineering, Data Pipelines, ETL, ELT, Terraform, Infrastructure as Code, IaC, CI/CD, Azure DevOps, GitHub Actions, Git, Python, SQL, Microsoft Azure, AWS, Google Cloud Platform, Cloud Infrastructure, Cloud Automation, Cloud Security, IAM, Data Governance, Data Lakes, Data Warehousing, Machine Learning Infrastructure, MLOps, Forecasting, Demand Planning, Supply Chain Analytics, Performance Optimization, FinOps, Cloud Cost Optimization, Platform Administration, DevOps, Infrastructure Automation

#LI-MM1 #LI-REMOTE