Databricks Platform Engineer

KPG99 INC; Quincy (MA); Hybrid; USD 140.000 - 190.000; Vollzeit

Zusammenfassung

KPG99 INC is seeking a Senior Platform Engineer for a Hybrid role in Quincy, MA. You will design, deploy, and manage Databricks workspaces across multiple Azure environments, implementing IaC with Terraform and ensuring security and governance across platforms.

You will build CI/CD pipelines with Azure DevOps, automate testing of data pipelines, and mentor teammates on Databricks best practices. In-person interview is required; this is a full-time direct hire.

Qualifikationen

• 7+ years of experience in platform engineering, infrastructure engineering, or DevOps.
• Strong expertise with Databricks infrastructure setup or Spark/Delta Lake environments.
• 3+ years of experience with Azure cloud services and Azure DevOps Pipelines, Repos, Artifacts.

Aufgaben

• Design, deploy, and manage Databricks workspaces across Azure subscriptions and environments (dev, staging, production).
• Implement IaC solutions using Terraform for Databricks resources.
• Establish security, including RBAC, network isolation, and data governance.
• Configure cluster management, auto-scaling, and cost optimization.
• Manage workspace connectivity and networking (VNets, private endpoints, firewalls).
• Build CI/CD pipelines using Azure DevOps for notebooks, jobs, and configurations.
• Automate testing and validation of data pipelines and create self-service deployment frameworks.
• Establish monitoring, alerting, and logging for Databricks environments.
• Automate workspace provisioning/deprovisioning; mentor team on best practices.

Kenntnisse

Infrastructure engineering

Platform engineering

Security best practices

Ausbildung

Bachelor of Computer Science

Tools

Terraform Enterprise

Azure DevOps

Databricks

Docker

Kubernetes

Jobbeschreibung

Hybrid at Quincy, MA

Full Time Direct Hire Role

In-Person Interview is Mandatory

Job Description

Must Have Technical/Functional Skills

Key Responsibilities

Platform & Infrastructure

• Design, deploy, and manage Databricks workspaces across multiple Azure subscriptions and environments (dev, staging, production)
• Implement Infrastructure-as-Code (IaC) solutions using Terraform for Databricks resources
• Establish and maintain security best practices, including network isolation, RBAC, and data governance
• Configure and optimize cluster management, including auto-scaling policies and cost optimization strategies
• Manage Databricks workspace connectivity and networking (VNets, private endpoints, firewalls)

DevOps & Automation

• Build CI/CD pipelines using Azure DevOps for deploying Databricks notebooks, jobs, and configurations
• Implement automated testing and validation of data pipelines
• Create self-service deployment frameworks and templates for development teams
• Establish monitoring, alerting, and logging solutions for Databricks environments
• Automate workspace provisioning and deprovisioning processes
• Partner with data engineers and analysts to understand platform requirements
• Provide technical guidance and documentation for platform usage
• Develop runbooks and operational procedures for incident response
• Mentor team members on Databricks best practices and Azure DevOps tools

Governance & Compliance

• Implement access controls and audit logging across Databricks workspaces
• Ensure compliance with organizational security and data governance policies
• Monitor and manage workspace usage, costs, and resource utilization
• Document architecture decisions and maintain platform documentation

Required Qualifications

• 7+ years of experience in platform engineering, infrastructure engineering, or DevOps
• 2+ years of hands-on experience with Databricks infrastructure setup (or Apache Spark/Delta Lake environments)
• 3+ years of experience with Azure cloud services and infrastructure
• 2+ years of experience with Azure DevOps (Pipelines, Repos, Artifacts)
• Strong proficiency in Infrastructure-as-Code tools (Terraform Enterprise)
• Solid understanding of CI/CD principles and implementation
• Experience with Linux/Unix environments and scripting (Bash, Python, or similar)
• Familiarity with containerization technologies (Docker, Kubernetes)
• Strong knowledge of networking, security, and identity management in cloud environments

Qualifications

• BACHELOR OF COMPUTER SCIENCE